And Vijay Says…

Everyone seems to agree on the same story: agents write the code, and the tools we already use stick around. Jira orchestrates the agents, GitHub verifies their work, Terraform enforces policy.

I buy the first half. The second half assumes that because a function survives, the vendor performing it keeps getting paid. That doesn’t follow. Teams will keep needing coordination, version control and infrastructure management. Whether they keep paying today’s prices for them is a separate question.

Start with Jira. It was built to organize human work, and it’s priced that way, by the seat. If an agent finds a bug, fixes it, runs the tests and prepares the release, a ticket may still exist, but fewer people touch it. The workflow survives, but the relationship between work performed and seats sold starts to break down.

Atlassian sees this. Rovo now meters AI activity and access to organizational context through credits, with additional usage billing scheduled to begin in December 2026. GitHub is moving toward a mix of seats and usage too. That’s adaptation, not collapse.

I still think Jira’s traditional economics are more exposed than GitHub’s, and the reason is margin. A traditional software seat costs relatively little to serve. Agent usage carries inference costs that someone has to pay. Usage revenue can replace seat revenue and still leave a worse business, so margin is the number to watch, not just revenue growth.

The best argument against me is that every agent becomes a heavy consumer of Atlassian’s organizational context. If that consumption carries attractive margins, AI could strengthen Jira. My bet is that inference-heavy usage will struggle to match the margins of traditional seats. Atlassian could prove me wrong if its context services command enough pricing power to offset those costs.

GitHub sits closer to the artifact. Repositories, history and provenance matter whoever writes the code. Its problem is different: passing tests isn’t the same as proving the change is correct.

An agent can write the code and the tests and still be wrong about the requirement. A second agent reviewing the work may share the first one’s assumptions.

Imagine a claims system. An agent finds that removing a validation step speeds processing by 35%. The tests pass and the security scan is clean, but that validation enforced an underwriting rule. The agent saw an inefficiency where the business had a constraint.

More tests derived from the same incomplete specification won’t fix that. The underwriting constraint has to be represented independently of the agent’s optimization objective.

Infrastructure has a version of the same problem. Terraform was declarative long before AI. What changes is volume. Ten changes that each pass policy can still add up to a capacity or resilience problem no single check sees.

I expect basic configuration generation and routine policy checks to lose differentiation, with pricing power moving toward whoever manages infrastructure state and cross-system consequences. More agents could also mean more managed resources, which helps incumbents. I’m predicting where pricing power concentrates, not who disappears.

In Migrating Scarcity, I argued that when something scarce gets cheap, the constraint moves to whatever it depended on.

Implementation was the scarce thing in software. As it gets cheaper, deciding what’s worth building and which risks to accept become relatively more important. Some of that work will be automated too.

Authority is the part I think holds, because someone has to answer for it. Regulators, insurers and boards can ask who allowed a consequential change. An agent can enforce a policy and propose a new one, but it shouldn’t be able to give itself permission to remove the constraint it was meant to respect.

Permission to execute is not permission to redefine success.

Scarce isn’t the same as profitable, though. Companies can meet accountability obligations through internal policies, contracts and existing controls. Audit logs, identity systems and attestations already cover part of the problem.

My bet is that a market still emerges around enforcing authority across systems. A repository can prove what changed. An infrastructure tool can prove what was deployed. Neither necessarily knows whether the change broke a business rule maintained in a third system.

I’d guess the winners are platforms that already control permissions: identity and security vendors, and GitHub where authority attaches to code. Atlassian has valuable context, but I suspect context will be easier to bundle than control. Cloud platforms are the wild card.

I’m wrong if agent consumption of Atlassian’s context grows enough to offset lost seats at attractive margins. I’m wrong if companies can delegate consequential decisions through existing automated policies without paying for another governance layer. And I’m wrong about the business, though not necessarily the constraint, if incumbents bundle all of this and charge nothing extra.

One more worry. If agents do the routine work, how do junior engineers build the judgment to own the big decisions later? Apprenticeship meant doing ordinary work first. A company could ship far more software while gradually losing the ability to govern it.

That would be a strange productivity victory. So the question isn’t which tools survive. It’s which constraints stay scarce enough that companies will pay to manage them.

I came across a paper on harness learning that got me thinking about something we don’t discuss enough in enterprise AI. We spend a lot of time worrying about what agents are allowed to do. What happens when they can also change how they do it?

The researchers train a model, called a proposer, to modify the software surrounding another model. This software, usually called the harness, determines how the model uses tools, handles information and carries out tasks. Rather than improving the underlying model, the proposer experiments with the harness and uses execution feedback to find changes that improve performance.

The interesting finding is that the proposer learns something transferable. It can improve harnesses for tasks it hasn’t previously encountered, without changing the underlying model’s weights.

I can see why people are excited about this. It suggests there may be more room to improve AI outcomes without constantly upgrading models. But it also raises a problem for enterprises. What if we eventually become much better at generating improvements than we are at evaluating and authorizing them?

When better performance isn’t enough

Consider an insurance company that uses an agent to settle straightforward claims below $10,000. The workflow includes policy verification, document checks, fraud screening and escalation rules. Some of those checks rarely affect the final settlement decision, but the company performs them because it needs evidence that the correct process was followed.

Now suppose a proposer discovers that removing one of those checks improves throughput by 20% without materially changing settlement accuracy. Its evaluation says the modified harness is better.

The difficulty is that the check may exist for reasons the evaluation never measured. An insurer may need those records to defend a disputed claim years later. A regulator may require them. Or the check may catch a rare class of errors that the test data barely represents.

The proposer has found a way to improve the metric it was given. It doesn’t need to be trying to circumvent a control. The control may simply look like unnecessary work when judged against an incomplete objective.

A change can improve measured performance while remaining unauthorized.

This example is hypothetical. The paper demonstrates improvements on reasoning and question-answering benchmarks, not autonomous modification of regulated insurance workflows. But the distinction it raises matters well beyond insurance.

Permission to execute versus permission to change

Harness learning makes an old distinction considerably more important: permission to perform an action is not the same as permission to change the system that authorizes and performs it.

In my book, Migrating Scarcity, I argue that when technology makes one capability abundant, the constraint often moves somewhere else. One idea I develop there is permission architecture: the authority, evidence and controls that determine whether an AI system may take an action. Harness learning extends that idea one level up.

An agent might have authority to settle a claim for $8,000. That doesn’t give it authority to eliminate a fraud check, change the escalation criteria or rewrite the process by which settlement decisions are approved.

Of course, enterprises already have change management. Banks have model risk management. Technology organizations have release approvals, separation of duties and production controls. None of this was invented because AI came along.

What changes is the rate at which modifications can be proposed, and the possibility that they affect the very controls governing the system. The existing principles still apply. The challenge is enforcing them when software can repeatedly propose changes to other software, potentially much faster than people can understand the consequences.

What should the proposer be allowed to change?

I think the architecture needs to separate the parts of the harness that can be optimized from the controls that govern its operation.

In the insurance example, settlement limits, mandatory documentation, fraud checks and escalation requirements would be identified as protected controls. Business and risk owners would decide which controls belong on that list, while technology would enforce the restrictions. The list would need regular review, because a control nobody remembered to protect is still a control.

The proposer could experiment with retrieval methods, context assembly and other parts of the workflow. Before any revision entered production, an independent deployment process would examine the changes, test their effects and determine whether they fell within previously delegated authority.

If the proposer removes a protected document check, the release gets blocked. It can still recommend the change and provide evidence that the existing requirement is too expensive or that another control would work better. But it cannot grant itself authority to implement the recommendation.

A change that never directly touches a protected control may still undermine it by altering the information it receives or the order in which steps execute. Code comparisons alone won’t be sufficient. The company needs behavioral testing, independent validation, versioned releases and a way to reconstruct which harness and policies governed a particular transaction.

There is also a problem with the human reviewer. A proposer that becomes good at finding improvements may also become good at explaining why those improvements deserve approval. Over time, risk owners could become rubber stamps for recommendations they cannot independently assess. We need reviewers with evidence they can verify themselves, enough time to examine consequential changes and the authority to say no.

Audit trails are necessary, but they don’t solve authorization. Suppose an unauthorized change removes a legally required document check and 50,000 claims are processed before anyone notices. Perfect logs would help reconstruct the incident and identify every affected transaction. They wouldn’t make the change authorized or the missing checks acceptable.

Human over the loop

We often talk about human in the loop, where someone approves individual decisions, and human on the loop, where people monitor automated execution and intervene when something goes wrong.

Harness learning introduces a different responsibility. If the process itself can change, someone must govern what those changes are allowed to be. That’s what I mean by human over the loop.

This requires more than an operations manager watching an exception dashboard. The people governing these changes need to understand technology, business processes, risk and how a seemingly local optimization might affect the broader operation. It is more demanding work than reviewing transactions, and enterprises will need to staff and organize for it.

That doesn’t mean every harness modification needs a committee meeting. Enterprises can preauthorize categories of changes that meet defined conditions and tests. But changes affecting regulatory obligations, financial authority or customer rights deserve a different level of scrutiny.

The economics of permission

If harness learning advances, the cost of discovering possible workflow improvements could fall substantially. The cost of validating those improvements, understanding their broader consequences and authorizing them may not fall nearly as quickly.

This is how scarcity migrates. If generating improvements becomes cheap and fast, the ability to evaluate and authorize them may become the binding constraint. Enterprises could discover that their capacity to govern change, rather than their capacity to generate it, determines how much AI productivity reaches production.

We are beginning to build systems that can propose changes to the way they operate. The research is still early, but the architecture for authorizing those changes can’t wait for it to mature.

AI may learn how to improve the system. The enterprise still has to decide what the system is allowed to become.

I’m increasingly skeptical of the assumption that progress in AI reasoning puts us on a path to superintelligence.

The expectation of far more powerful AI isn’t imaginary. In his October 2024 essay Machines of Loving Grace, Anthropic’s Dario Amodei described the possibility of a “country of geniuses in a datacenter.” He thought such powerful AI could arrive as early as 2026, though he acknowledged it might take considerably longer.

I can understand the optimism. Reasoning models have improved enormously. They solve difficult mathematical problems, write and debug code, and find solutions that would take human experts considerable time. Give them more opportunities to search, test and refine their answers, and performance often improves.

I expect that progress to continue. What I question is whether extending today’s methods gets us to machines that substantially exceed the best human minds across a broad range of intellectual tasks.

I think we’re heading toward something enormously valuable anyway: hyperautomation. I use the term in its established sense of combining AI with conventional automation to handle increasingly complex business processes. The economic case for doing this doesn’t depend on superintelligence.

Search needs a scorekeeper

A significant part of recent progress has come from reinforcement learning and additional computation devoted to finding better answers. Models generate possible solutions, receive feedback and improve.

DeepSeek-R1, subsequently published in Nature, is a good example. Its researchers used rule-based rewards for verifiable reasoning tasks, avoiding neural reward models in that setting because of their vulnerability to reward hacking during large-scale reinforcement learning.

This works remarkably well when evaluation is reliable. Mathematical answers can be checked. Code can be compiled and tested. Algorithms can be measured against known objectives.

But code that passes its tests may still solve the wrong problem. An algorithm can optimize its assigned metric while making the overall system worse.

Another model can judge the output, but that introduces its own difficulties. Research by Gao, Schulman and Hilton demonstrated that pushing optimization too aggressively against an imperfect reward model can eventually degrade performance against the underlying objective.

The harder problem really isn’t generating more candidate answers – rather it’s in figuring out which ones are good when the criteria are uncertain.

But AI is already making discoveries

The counterexamples deserve serious attention.

AlphaFold made an extraordinary advance in predicting protein structures. It learned from experimentally determined structures, giving it a foundation of physical evidence from which to develop predictions. Its confidence estimates are useful, although they don’t guarantee correctness.

AlphaEvolve is even more interesting. Google’s system found a method to multiply 4×4 complex matrices using 48 scalar multiplications, improving on a longstanding result. It also designed components of a gradient-based optimization procedure used to discover matrix multiplication algorithms.

These are genuine advances, and I absolutely expect many more. I am quite excited about what else will come out from this direction !

But AlphaEvolve also illustrates the importance of evaluation. It generates programs and tests them against measurable objectives. Google describes the system as applicable to problems whose solutions can be expressed as algorithms and automatically verified.

That’s a powerful method for discovery. It doesn’t establish that the system can independently recognize when an entire scientific framework is inadequate, develop an alternative and determine how to validate it.

Humans don’t begin with perfect scorekeepers either. Scientists invent experiments precisely because existing evidence is insufficient. AI may eventually become exceptionally good at this. I just haven’t seen convincing evidence that the current approach reliably generalizes to such problems.

Berkeley’s A-Lab offers another example. Its autonomous laboratory demonstrated impressive materials synthesis, but subsequent scrutiny raised questions about material identification and novelty. In a January 2026 correction, the authors clarified that the materials described as new were new to their prediction platform, not necessarily new to science. Their reanalysis confirmed 36 of 40 reported successes, with four inconclusive.

The automation was real. Establishing exactly what had been discovered required additional scrutiny.

Hyperautomation is already a big enough prize

This distinction matters in enterprise AI, where I spend much of my time.

Consider accounts payable. An agent can extract invoice data, match purchase orders, check receipts, apply payment policies, identify exceptions and initiate approved payments. Much of this works because the checks are deterministic. The invoice matches or it doesn’t. The payment falls within policy or it doesn’t. The transaction completes or fails.

The important design question is often not how intelligent the agent is, but what tells it when it’s wrong.

Move into a disputed invoice involving a strategic supplier, and the evaluation becomes less straightforward. Should the company enforce the contract, preserve the relationship or renegotiate the terms? An AI system may eventually make many of those decisions too, but it needs some way to resolve competing objectives and establish its authority to act.

We can automate an extraordinary amount of work without solving every such problem. Claims processing, financial close, customer service and software development can all benefit from combinations of reasoning models, conventional software, classifiers and human oversight. Not every task needs an LLM, much less a superintelligent one.

There are difficult engineering and organizational problems to solve. Integration, security, context, evaluation and process redesign remain substantial challenges. But addressing them could transform enterprise economics even if superintelligence never arrives.

This fits the argument I made in Migrating Scarcity. As execution becomes cheaper, the constraint moves toward specifying objectives, evaluating outcomes and deciding who or what has authority to act. Better automation doesn’t eliminate those questions. It makes them more important.

What would change my mind?

I’d take the current path to superintelligence more seriously if AI systems began independently identifying important scientific problems, constructing new explanations and designing experiments whose novel predictions were subsequently confirmed by independent researchers.

I’d also look for systems that recognize flaws in their own evaluation methods and develop better ones. A meaningful demonstration would be an AI discovering that its existing test rewarded the wrong behavior, designing a replacement, and showing that the replacement predicts independently measured outcomes more accurately on unfamiliar problems. Repeating that across different domains would be much stronger evidence than improvement on another benchmark.

I don’t know whether superintelligence is possible or whether some extension of today’s methods will eventually produce it. Scaling has surprised us before. But possibility isn’t evidence, and progress in search and automation doesn’t by itself establish a path to superintelligence.

I think hyperautomation will change the economics of entire industries. We have barely begun capturing that opportunity, and the engineering work ahead is substantial.

We don’t need to call it superintelligence to recognize how valuable it could become.