I’m increasingly skeptical of the assumption that progress in AI reasoning puts us on a path to superintelligence.
The expectation of far more powerful AI isn’t imaginary. In his October 2024 essay Machines of Loving Grace, Anthropic’s Dario Amodei described the possibility of a “country of geniuses in a datacenter.” He thought such powerful AI could arrive as early as 2026, though he acknowledged it might take considerably longer.
I can understand the optimism. Reasoning models have improved enormously. They solve difficult mathematical problems, write and debug code, and find solutions that would take human experts considerable time. Give them more opportunities to search, test and refine their answers, and performance often improves.
I expect that progress to continue. What I question is whether extending today’s methods gets us to machines that substantially exceed the best human minds across a broad range of intellectual tasks.
I think we’re heading toward something enormously valuable anyway: hyperautomation. I use the term in its established sense of combining AI with conventional automation to handle increasingly complex business processes. The economic case for doing this doesn’t depend on superintelligence.
Search needs a scorekeeper
A significant part of recent progress has come from reinforcement learning and additional computation devoted to finding better answers. Models generate possible solutions, receive feedback and improve.
DeepSeek-R1, subsequently published in Nature, is a good example. Its researchers used rule-based rewards for verifiable reasoning tasks, avoiding neural reward models in that setting because of their vulnerability to reward hacking during large-scale reinforcement learning.
This works remarkably well when evaluation is reliable. Mathematical answers can be checked. Code can be compiled and tested. Algorithms can be measured against known objectives.
But code that passes its tests may still solve the wrong problem. An algorithm can optimize its assigned metric while making the overall system worse.
Another model can judge the output, but that introduces its own difficulties. Research by Gao, Schulman and Hilton demonstrated that pushing optimization too aggressively against an imperfect reward model can eventually degrade performance against the underlying objective.
The harder problem really isn’t generating more candidate answers – rather it’s in figuring out which ones are good when the criteria are uncertain.
But AI is already making discoveries
The counterexamples deserve serious attention.
AlphaFold made an extraordinary advance in predicting protein structures. It learned from experimentally determined structures, giving it a foundation of physical evidence from which to develop predictions. Its confidence estimates are useful, although they don’t guarantee correctness.
AlphaEvolve is even more interesting. Google’s system found a method to multiply 4×4 complex matrices using 48 scalar multiplications, improving on a longstanding result. It also designed components of a gradient-based optimization procedure used to discover matrix multiplication algorithms.
These are genuine advances, and I absolutely expect many more. I am quite excited about what else will come out from this direction !
But AlphaEvolve also illustrates the importance of evaluation. It generates programs and tests them against measurable objectives. Google describes the system as applicable to problems whose solutions can be expressed as algorithms and automatically verified.
That’s a powerful method for discovery. It doesn’t establish that the system can independently recognize when an entire scientific framework is inadequate, develop an alternative and determine how to validate it.
Humans don’t begin with perfect scorekeepers either. Scientists invent experiments precisely because existing evidence is insufficient. AI may eventually become exceptionally good at this. I just haven’t seen convincing evidence that the current approach reliably generalizes to such problems.
Berkeley’s A-Lab offers another example. Its autonomous laboratory demonstrated impressive materials synthesis, but subsequent scrutiny raised questions about material identification and novelty. In a January 2026 correction, the authors clarified that the materials described as new were new to their prediction platform, not necessarily new to science. Their reanalysis confirmed 36 of 40 reported successes, with four inconclusive.
The automation was real. Establishing exactly what had been discovered required additional scrutiny.
Hyperautomation is already a big enough prize
This distinction matters in enterprise AI, where I spend much of my time.
Consider accounts payable. An agent can extract invoice data, match purchase orders, check receipts, apply payment policies, identify exceptions and initiate approved payments. Much of this works because the checks are deterministic. The invoice matches or it doesn’t. The payment falls within policy or it doesn’t. The transaction completes or fails.
The important design question is often not how intelligent the agent is, but what tells it when it’s wrong.
Move into a disputed invoice involving a strategic supplier, and the evaluation becomes less straightforward. Should the company enforce the contract, preserve the relationship or renegotiate the terms? An AI system may eventually make many of those decisions too, but it needs some way to resolve competing objectives and establish its authority to act.
We can automate an extraordinary amount of work without solving every such problem. Claims processing, financial close, customer service and software development can all benefit from combinations of reasoning models, conventional software, classifiers and human oversight. Not every task needs an LLM, much less a superintelligent one.
There are difficult engineering and organizational problems to solve. Integration, security, context, evaluation and process redesign remain substantial challenges. But addressing them could transform enterprise economics even if superintelligence never arrives.
This fits the argument I made in Migrating Scarcity. As execution becomes cheaper, the constraint moves toward specifying objectives, evaluating outcomes and deciding who or what has authority to act. Better automation doesn’t eliminate those questions. It makes them more important.
What would change my mind?
I’d take the current path to superintelligence more seriously if AI systems began independently identifying important scientific problems, constructing new explanations and designing experiments whose novel predictions were subsequently confirmed by independent researchers.
I’d also look for systems that recognize flaws in their own evaluation methods and develop better ones. A meaningful demonstration would be an AI discovering that its existing test rewarded the wrong behavior, designing a replacement, and showing that the replacement predicts independently measured outcomes more accurately on unfamiliar problems. Repeating that across different domains would be much stronger evidence than improvement on another benchmark.
I don’t know whether superintelligence is possible or whether some extension of today’s methods will eventually produce it. Scaling has surprised us before. But possibility isn’t evidence, and progress in search and automation doesn’t by itself establish a path to superintelligence.
I think hyperautomation will change the economics of entire industries. We have barely begun capturing that opportunity, and the engineering work ahead is substantial.
We don’t need to call it superintelligence to recognize how valuable it could become.
