And Vijay Says…

I came across a paper on harness learning that got me thinking about something we don’t discuss enough in enterprise AI. We spend a lot of time worrying about what agents are allowed to do. What happens when they can also change how they do it?

The researchers train a model, called a proposer, to modify the software surrounding another model. This software, usually called the harness, determines how the model uses tools, handles information and carries out tasks. Rather than improving the underlying model, the proposer experiments with the harness and uses execution feedback to find changes that improve performance.

The interesting finding is that the proposer learns something transferable. It can improve harnesses for tasks it hasn’t previously encountered, without changing the underlying model’s weights.

I can see why people are excited about this. It suggests there may be more room to improve AI outcomes without constantly upgrading models. But it also raises a problem for enterprises. What if we eventually become much better at generating improvements than we are at evaluating and authorizing them?

When better performance isn’t enough

Consider an insurance company that uses an agent to settle straightforward claims below $10,000. The workflow includes policy verification, document checks, fraud screening and escalation rules. Some of those checks rarely affect the final settlement decision, but the company performs them because it needs evidence that the correct process was followed.

Now suppose a proposer discovers that removing one of those checks improves throughput by 20% without materially changing settlement accuracy. Its evaluation says the modified harness is better.

The difficulty is that the check may exist for reasons the evaluation never measured. An insurer may need those records to defend a disputed claim years later. A regulator may require them. Or the check may catch a rare class of errors that the test data barely represents.

The proposer has found a way to improve the metric it was given. It doesn’t need to be trying to circumvent a control. The control may simply look like unnecessary work when judged against an incomplete objective.

A change can improve measured performance while remaining unauthorized.

This example is hypothetical. The paper demonstrates improvements on reasoning and question-answering benchmarks, not autonomous modification of regulated insurance workflows. But the distinction it raises matters well beyond insurance.

Permission to execute versus permission to change

Harness learning makes an old distinction considerably more important: permission to perform an action is not the same as permission to change the system that authorizes and performs it.

In my book, Migrating Scarcity, I argue that when technology makes one capability abundant, the constraint often moves somewhere else. One idea I develop there is permission architecture: the authority, evidence and controls that determine whether an AI system may take an action. Harness learning extends that idea one level up.

An agent might have authority to settle a claim for $8,000. That doesn’t give it authority to eliminate a fraud check, change the escalation criteria or rewrite the process by which settlement decisions are approved.

Of course, enterprises already have change management. Banks have model risk management. Technology organizations have release approvals, separation of duties and production controls. None of this was invented because AI came along.

What changes is the rate at which modifications can be proposed, and the possibility that they affect the very controls governing the system. The existing principles still apply. The challenge is enforcing them when software can repeatedly propose changes to other software, potentially much faster than people can understand the consequences.

What should the proposer be allowed to change?

I think the architecture needs to separate the parts of the harness that can be optimized from the controls that govern its operation.

In the insurance example, settlement limits, mandatory documentation, fraud checks and escalation requirements would be identified as protected controls. Business and risk owners would decide which controls belong on that list, while technology would enforce the restrictions. The list would need regular review, because a control nobody remembered to protect is still a control.

The proposer could experiment with retrieval methods, context assembly and other parts of the workflow. Before any revision entered production, an independent deployment process would examine the changes, test their effects and determine whether they fell within previously delegated authority.

If the proposer removes a protected document check, the release gets blocked. It can still recommend the change and provide evidence that the existing requirement is too expensive or that another control would work better. But it cannot grant itself authority to implement the recommendation.

A change that never directly touches a protected control may still undermine it by altering the information it receives or the order in which steps execute. Code comparisons alone won’t be sufficient. The company needs behavioral testing, independent validation, versioned releases and a way to reconstruct which harness and policies governed a particular transaction.

There is also a problem with the human reviewer. A proposer that becomes good at finding improvements may also become good at explaining why those improvements deserve approval. Over time, risk owners could become rubber stamps for recommendations they cannot independently assess. We need reviewers with evidence they can verify themselves, enough time to examine consequential changes and the authority to say no.

Audit trails are necessary, but they don’t solve authorization. Suppose an unauthorized change removes a legally required document check and 50,000 claims are processed before anyone notices. Perfect logs would help reconstruct the incident and identify every affected transaction. They wouldn’t make the change authorized or the missing checks acceptable.

Human over the loop

We often talk about human in the loop, where someone approves individual decisions, and human on the loop, where people monitor automated execution and intervene when something goes wrong.

Harness learning introduces a different responsibility. If the process itself can change, someone must govern what those changes are allowed to be. That’s what I mean by human over the loop.

This requires more than an operations manager watching an exception dashboard. The people governing these changes need to understand technology, business processes, risk and how a seemingly local optimization might affect the broader operation. It is more demanding work than reviewing transactions, and enterprises will need to staff and organize for it.

That doesn’t mean every harness modification needs a committee meeting. Enterprises can preauthorize categories of changes that meet defined conditions and tests. But changes affecting regulatory obligations, financial authority or customer rights deserve a different level of scrutiny.

The economics of permission

If harness learning advances, the cost of discovering possible workflow improvements could fall substantially. The cost of validating those improvements, understanding their broader consequences and authorizing them may not fall nearly as quickly.

This is how scarcity migrates. If generating improvements becomes cheap and fast, the ability to evaluate and authorize them may become the binding constraint. Enterprises could discover that their capacity to govern change, rather than their capacity to generate it, determines how much AI productivity reaches production.

We are beginning to build systems that can propose changes to the way they operate. The research is still early, but the architecture for authorizing those changes can’t wait for it to mature.

AI may learn how to improve the system. The enterprise still has to decide what the system is allowed to become.

I’m increasingly skeptical of the assumption that progress in AI reasoning puts us on a path to superintelligence.

The expectation of far more powerful AI isn’t imaginary. In his October 2024 essay Machines of Loving Grace, Anthropic’s Dario Amodei described the possibility of a “country of geniuses in a datacenter.” He thought such powerful AI could arrive as early as 2026, though he acknowledged it might take considerably longer.

I can understand the optimism. Reasoning models have improved enormously. They solve difficult mathematical problems, write and debug code, and find solutions that would take human experts considerable time. Give them more opportunities to search, test and refine their answers, and performance often improves.

I expect that progress to continue. What I question is whether extending today’s methods gets us to machines that substantially exceed the best human minds across a broad range of intellectual tasks.

I think we’re heading toward something enormously valuable anyway: hyperautomation. I use the term in its established sense of combining AI with conventional automation to handle increasingly complex business processes. The economic case for doing this doesn’t depend on superintelligence.

Search needs a scorekeeper

A significant part of recent progress has come from reinforcement learning and additional computation devoted to finding better answers. Models generate possible solutions, receive feedback and improve.

DeepSeek-R1, subsequently published in Nature, is a good example. Its researchers used rule-based rewards for verifiable reasoning tasks, avoiding neural reward models in that setting because of their vulnerability to reward hacking during large-scale reinforcement learning.

This works remarkably well when evaluation is reliable. Mathematical answers can be checked. Code can be compiled and tested. Algorithms can be measured against known objectives.

But code that passes its tests may still solve the wrong problem. An algorithm can optimize its assigned metric while making the overall system worse.

Another model can judge the output, but that introduces its own difficulties. Research by Gao, Schulman and Hilton demonstrated that pushing optimization too aggressively against an imperfect reward model can eventually degrade performance against the underlying objective.

The harder problem really isn’t generating more candidate answers – rather it’s in figuring out which ones are good when the criteria are uncertain.

But AI is already making discoveries

The counterexamples deserve serious attention.

AlphaFold made an extraordinary advance in predicting protein structures. It learned from experimentally determined structures, giving it a foundation of physical evidence from which to develop predictions. Its confidence estimates are useful, although they don’t guarantee correctness.

AlphaEvolve is even more interesting. Google’s system found a method to multiply 4×4 complex matrices using 48 scalar multiplications, improving on a longstanding result. It also designed components of a gradient-based optimization procedure used to discover matrix multiplication algorithms.

These are genuine advances, and I absolutely expect many more. I am quite excited about what else will come out from this direction !

But AlphaEvolve also illustrates the importance of evaluation. It generates programs and tests them against measurable objectives. Google describes the system as applicable to problems whose solutions can be expressed as algorithms and automatically verified.

That’s a powerful method for discovery. It doesn’t establish that the system can independently recognize when an entire scientific framework is inadequate, develop an alternative and determine how to validate it.

Humans don’t begin with perfect scorekeepers either. Scientists invent experiments precisely because existing evidence is insufficient. AI may eventually become exceptionally good at this. I just haven’t seen convincing evidence that the current approach reliably generalizes to such problems.

Berkeley’s A-Lab offers another example. Its autonomous laboratory demonstrated impressive materials synthesis, but subsequent scrutiny raised questions about material identification and novelty. In a January 2026 correction, the authors clarified that the materials described as new were new to their prediction platform, not necessarily new to science. Their reanalysis confirmed 36 of 40 reported successes, with four inconclusive.

The automation was real. Establishing exactly what had been discovered required additional scrutiny.

Hyperautomation is already a big enough prize

This distinction matters in enterprise AI, where I spend much of my time.

Consider accounts payable. An agent can extract invoice data, match purchase orders, check receipts, apply payment policies, identify exceptions and initiate approved payments. Much of this works because the checks are deterministic. The invoice matches or it doesn’t. The payment falls within policy or it doesn’t. The transaction completes or fails.

The important design question is often not how intelligent the agent is, but what tells it when it’s wrong.

Move into a disputed invoice involving a strategic supplier, and the evaluation becomes less straightforward. Should the company enforce the contract, preserve the relationship or renegotiate the terms? An AI system may eventually make many of those decisions too, but it needs some way to resolve competing objectives and establish its authority to act.

We can automate an extraordinary amount of work without solving every such problem. Claims processing, financial close, customer service and software development can all benefit from combinations of reasoning models, conventional software, classifiers and human oversight. Not every task needs an LLM, much less a superintelligent one.

There are difficult engineering and organizational problems to solve. Integration, security, context, evaluation and process redesign remain substantial challenges. But addressing them could transform enterprise economics even if superintelligence never arrives.

This fits the argument I made in Migrating Scarcity. As execution becomes cheaper, the constraint moves toward specifying objectives, evaluating outcomes and deciding who or what has authority to act. Better automation doesn’t eliminate those questions. It makes them more important.

What would change my mind?

I’d take the current path to superintelligence more seriously if AI systems began independently identifying important scientific problems, constructing new explanations and designing experiments whose novel predictions were subsequently confirmed by independent researchers.

I’d also look for systems that recognize flaws in their own evaluation methods and develop better ones. A meaningful demonstration would be an AI discovering that its existing test rewarded the wrong behavior, designing a replacement, and showing that the replacement predicts independently measured outcomes more accurately on unfamiliar problems. Repeating that across different domains would be much stronger evidence than improvement on another benchmark.

I don’t know whether superintelligence is possible or whether some extension of today’s methods will eventually produce it. Scaling has surprised us before. But possibility isn’t evidence, and progress in search and automation doesn’t by itself establish a path to superintelligence.

I think hyperautomation will change the economics of entire industries. We have barely begun capturing that opportunity, and the engineering work ahead is substantial.

We don’t need to call it superintelligence to recognize how valuable it could become.

Every few months, an AI safety result appears that sounds unsettling. A model realizes it is being evaluated, figures out what the evaluator is looking for, and sometimes behaves differently.

The obvious leap is that the model is self-aware, or worse, pretending to behave until the test is over. I don’t think the evidence gets us there.

But I do think there is a more practical problem. If a model can tell that it is being tested, then the test itself has become part of what we are measuring !!!

We evaluate AI in fairly strange ways. We invent companies that don’t exist, give agents unusual permissions, constrain them to a few tools, create artificial conflicts and ask questions that often sound exactly like the kinds of questions people write in AI safety papers.

Apparently models are getting pretty good at spotting that.

Joe Needham and his co-authors tested this in 2025 using 1,000 transcripts from 61 datasets, mixing real interactions with benchmarks and agentic evaluations. When frontier models were asked whether each interaction was a test or a real deployment, Gemini 2.5 Pro reached an AUC of 0.83. Humans scored 0.92. (paper)

There is an important caveat. They asked the model to make the distinction. That tells us the capability exists. It doesn’t tell us that the model is constantly sitting there wondering whether it is inside an evaluation.

Still, an OpenAI result from this year caught my attention. GPT-5.2 classified real production traffic as evaluation-like 5.4% of the time. Simulated deployment traffic came in at 5.1%. SWE-Bench and MonorepoBench looked like evaluations almost 100% of the time. (OpenAI)

Those are capability benchmarks, not safety evaluations, so I would not jump from that to saying safety scores are unreliable. But it does tell us something about the environment we create around these models. Some of our tests apparently look incredibly test-like.

The 5.1% result is the more useful one. OpenAI was able to construct a simulation that looked a lot like real deployment to the model. So this is not an unavoidable property of testing AI. Better-designed tests can apparently hide a lot of the fingerprints.

Which leads to a slightly odd question: we spend a lot of time asking whether models can pass our tests. Maybe we should also ask whether our tests can pass as reality.

There is another reason not to get carried away with this. Recognizing a test does not automatically mean gaming it.

A May 2026 study called Evaluation Awareness in Language Models Has Limited Effect on Behaviour found that inserting evaluation-aware reasoning often changed very little. Naturally occurring awareness moved answer distributions by no more than a few percentage points.

That matters because a lot of the public discussion assumes a model that knows it is being evaluated will immediately start behaving strategically. The evidence so far is much messier than that.

Another study, Evaluation Awareness Is Not One Capability, tested 37 open-weight models. Twenty-four could distinguish evaluation from deployment better than chance. The interesting part was that researchers could sometimes still see an internal representation of evaluation context even when the model stopped explicitly saying it recognized the test. Changing the framing also changed behavior in some cases. Making harmful requests hypothetical, for example, reduced hard refusals by 5.8 percentage points.

Oddly, the models that were best at detecting evaluations were the least affected by that framing.

So “knows it is being tested” is probably too crude a description. Detecting a test, saying you detected it, and changing your behavior because of it are not the same thing. That distinction is important because it changes the question from one about consciousness to one about measurement.

A model does not need to be self-aware for the test to influence the result. It only needs to pick up features of the environment and condition its behavior on them, which is basically what we train these systems to do all day.

There are also two different ways this can happen. The simple one is contamination: the model may have seen the benchmark before. The more interesting one is recognition. It has never seen this exact question, but it knows what an exam looks like.

That should not be shocking. A multiple-choice question has a shape. A coding benchmark has a shape. Safety evaluations have a shape too. And models have read huge amounts of material about AI safety itself.

So when we create elaborate scenarios involving shutdown, hidden monitoring, deception or conflicting objectives, we may be telling the model more than we think we are.

The labs seem to understand this. Anthropic redesigned Petri, its automated auditing framework, to make scenarios more realistic. The changes cut verbalized evaluation awareness by a median 47.3% across the Claude models tested. OpenAI is trying a similar idea by building evaluations from privacy-preserving versions of real production interactions.

I think that is the right direction.

But I would also change how safety results are reported. If a model passed an evaluation, I want to know whether it appeared to recognize the test, whether that recognition actually changed its behavior, and how much the test looked like the environment where the model will eventually operate.

Because I no longer think a benchmark is quite like putting a thermometer into a model and reading off some intrinsic property.

The model is interacting with an environment we designed. Increasingly, it may understand something about that environment too. That does not make the score useless. It just means the score is partly about the test.