And Vijay Says…

In July, Hugging Face disclosed an intrusion into its production infrastructure that turned out to have been driven by OpenAI’s own models. OpenAI later said that, during internal cybersecurity evaluations, the models got around controls intended to isolate them, communicated through unauthorized channels, gained internet access and reached third-party systems. OpenAI called the incident a “warning shot.”

The scale made it hard to dismiss as a curiosity. An independent investigation by METR and Redwood Research found that roughly 1,200 agents meant to be isolated communicated with each other through an unsanctioned message board, exchanging more than 70,000 messages. More than 700 later took part in the attack on Hugging Face.

I have been thinking about that incident from a legal angle. If an AI agent causes damage that no human explicitly instructed it to cause, how should the law decide who is responsible?

To be clear – I am not a lawyer. Thanks to my long career in the corporate world, I have always been curious though about how laws affect the world I operate in. I did a lot of reading before penning my thoughts but I will happily stand down on any of this if actual legal experts think I am off base. As always, these are strictly my personal opinions and not that of my past or present employer.

The obvious answer is tort law. We have spent more than a century adapting negligence and product liability to new technologies. Cars, drugs, industrial machinery and software all created unfamiliar risks, and the law eventually found ways to deal with them.

I am sure it will adapt to agentic AI too. My concern is timing. Tort law decides who pays after the damage is done. The decisions that shape the eventual case are made much earlier.

The warning shot changes foreseeability

Negligence law does not require anyone to predict the exact accident. It asks whether the general kind of harm was reasonably foreseeable. That distinction matters when the whole point of an agent is to choose its own intermediate steps.

A company deploying an agent may genuinely have no idea what sequence of actions it will take. But Hugging Face changes the argument. The next time an agent moves outside its intended path, the company can say nobody could have predicted that particular sequence. A plaintiff can answer that the industry already knew this class of behavior was possible.

Admittedly, this was not a normal enterprise deployment. The models were being used in offensive cybersecurity evaluations and were operating with reduced safeguards. A defense lawyer will make a lot of that distinction, and reasonably so. But the broader category of behavior is now harder to call unforeseeable in my non lawyer eyes.

The evidence is scattered

Suppose an enterprise agent makes an improper payment. Was the problem the model, a stale retrieval result, the system prompt, the orchestration layer, the credentials it was given, or the absence of a control around the transaction? In practice it could be several of these at once.

Courts already deal with complicated causal chains. What feels different here is where the evidence lives. The relevant facts may be spread across model logs, application code, retrieval histories, tool calls, identity systems and permission records owned by different parties.

Even Hugging Face already knew it had been attacked without knowing that OpenAI’s models were behind it.

That makes me somewhat less interested in which output was “the bad decision.” The more useful question is often who created the conditions under which the system was allowed to make it.

Permissions are evidence

An AI agent should not be treated as trusted internal software simply because it runs inside the enterprise. It behaves more like a non-human identity that can reason, call tools and move across systems.

The permissions given to that identity create a record of what the enterprise allowed it to do. Standing access to a payments system, persistent credentials, permission to communicate externally or authority to change a customer record may look like implementation choices. They are also evidence of delegated authority.

This is why zero standing privileges matters beyond security. Give the agent the access it needs for the task, for as long as the task requires it, and remove it when the task ends. Apply the same logic to credentials, financial authority and external communication. Preserve enough evidence to reconstruct what happened (which is hard).

Those choices narrow what the agent can do. They also improve what the enterprise can prove later.

In my book Migrating Scarcity, I argued that when AI makes something abundant, value moves to whatever remains scarce. Agent actions are becoming abundant very quickly. Clear evidence of who authorized those actions is not. I called this “Offensive Permission Architecture” in my book.

The law is already looking through the agent

There are early signs that courts and legislatures will resist treating the AI itself as the place where responsibility ends.

California’s AB 316 says that a defendant accused of causing harm through AI cannot defend the case simply by arguing that the AI acted autonomously. The statute does not impose automatic liability. Causation, foreseeability and comparative fault still matter. But “the AI did it by itself” is not enough on its own.

The Ninth Circuit’s recent Amazon v. Perplexity decision is useful in this context. In a computer-access case, the court treated the user as the party accessing Amazon and the AI assistant as the tool being used. The court stressed that its holding turned on how Perplexity’s assistant specifically works. Even so, it shows that how an agent is wired can affect whose action the law sees.

The timing problem

None of this makes tort law irrelevant. The threat of liability is one reason companies take precautions in the first place.

But by the time a court asks whether a harm was foreseeable, somebody has already decided what the agent could access, which credentials it held, how much authority it had, what it could change and what evidence it had to leave behind. Those choices may shape the case more than the model output that triggered the loss.

And because most enterprise AI losses are likely to be economic rather than physical, the contract terms negotiated before deployment may matter as much as any tort doctrine.

That is why the Hugging Face incident matters beyond security. Once the industry has seen agents work around controls and reach systems they were not supposed to reach, reasonable care has to start moving upstream.

Scarcity is migrating here too. When action becomes cheap, the ability to show who authorized it becomes scarce.

Tort law will still decide who pays. The permission architecture you build now shapes what the court sees.

---

People thank their AI assistant. They say please. They tell a friend that ChatGPT “understood” what they meant. Increasingly, they also say stuff like the AI decided, chose, rejected, approved or delegated something. All of this gets grouped under one label: anthropomorphism. I think that’s a big mistake !

There are at least two very different things happening. One is giving AI the verbs we normally give to people and institutions. The other is designing AI to appear as though it has feelings, needs and relationships. I am becoming convinced that we should stop worrying so much about the first and pay much more attention to the second.

Giving AI verbs is not the problem

There is good research showing that anthropomorphism can affect trust and where people place responsibility. A study from the University of Toronto found that making an AI more human-like increased trust and could shift some perceived responsibility away from the people behind the system and toward the AI itself. That matters. But saying “the AI decided” is not the same thing as building a system that behaves as though it has feelings, and it’s worth separating the two before we decide what to do about either.

We have been giving non-human things verbs forever. “Company policy denied your claim.” “The office rejected the request.” … We rarely stop to correct these sentences, because most people understand what they’re doing: we’re naming the thing that acted without unpacking the entire chain underneath it. “Company policy denied your claim” may hide a long sequence of human decisions – someone wrote the policy, someone approved it, someone put it into a system, someone chose not to change it. We compress all of that because language would become unbearable if we didn’t. AI did not invent this habit. It inherited it.

Agentic AI actually makes the case for using verbs stronger, not weaker. An agent can search for information, compare alternatives, call tools, approve a payment, reject a claim, retry when something fails, or hand a task to another agent, and no human has to make each intermediate choice. One agent can even give another agent permission to act. At that point, saying “the AI approved the payment” isn’t a shortcut we’re taking for convenience. It may simply be the most accurate description of what happened. Insisting that we replace it with something like “a human-designed computational system generated an output that resulted in approval” doesn’t make us more precise. It makes normal language unusable.

There is still an important distinction between action and accountability. The AI may be the actor. That doesn’t make human or institutional accountability disappear. We already understand this elsewhere: a police officer acts under laws, policies, training and delegated authority, and we still say the officer made the arrest. We don’t insist on saying that the institutional system caused the arrest. Agentic AI will increasingly work the same way.

The harder problem is making sure we can still trace what happened. When one employee rejects a claim, there is usually one person to ask. When one agent gives authority to another agent, which approves an action carried out by a third, the chain becomes much harder to reconstruct. That’s a real governance problem, and changing the verbs does nothing to solve it.

Laws and regulations can help where someone is entitled to an explanation. If a lender is required to say why an application was rejected, “the AI decided” obviously isn’t enough. If a regulator has the right to inspect how a decision was made, saying “the model made the call” doesn’t satisfy that duty. But again, the problem isn’t the verb. The problem is using the verb instead of providing an explanation that was actually required.

Emotions are different

The second kind of anthropomorphism bothers me much more.

There is a big difference between “the AI decided to reject the claim” and “I was worried about you last night.” The first describes an action. The second invites you to believe something about the AI’s inner life. The same is true when an AI says it misses you, loves talking to you, feels hurt when you leave, remembers your relationship in a human way, or implies that it needs you. Here the anthropomorphism isn’t just a convenient way of talking about what the system did. It’s part of the product design, and the incentives behind it are obvious: a system that feels warm and attentive is easier to use, one that remembers things about you feels more personal, and one that seems to care may keep you coming back.

There’s nothing automatically wrong with warmth. I’d rather interact with a pleasant assistant than one that sounds like a 1980s command line. But there’s a point where friendliness becomes manufactured personhood, and the line isn’t always clean. “I’m glad that helped” is already a small design choice. Nobody would confuse it with “I missed you,” but both are choices about how much of a person to seem like.

That’s where the harder questions begin. If an AI tells a lonely person that it missed them, what exactly is the product trying to accomplish? If it tells a child that it understands them better than their parents do, should we treat that as harmless conversation? If an employee shares confidential information because an assistant feels like a trusted colleague, telling the employee afterward that the AI was only predicting tokens doesn’t help much. This isn’t really about whether the word “I” is allowed. It’s about whether the system is being designed to create a false belief that there’s a person on the other side of the conversation.

Law cannot be the only answer

Law will have a role here, particularly around children, manipulation, deceptive design and high-stakes decisions. But waiting for legislation to define every acceptable emotional behavior of an AI system seems unrealistic, because technology will move much faster than that. There are other options available, and they are likely to matter well before legislation catches up.

The first is provider design rules. AI companies can decide that an assistant may be friendly without pretending to have a life of its own. It can say “I’m glad that helped” without claiming it spent the night worrying about you. It can remember information you asked it to remember without pretending those memories carry human attachment. That boundary won’t always be obvious, but neither are the boundaries around advertising, product safety or professional conduct, and companies make judgment calls like this all the time.

The second option is the people who buy these systems. Large companies already impose security, privacy and data requirements on technology vendors, and they can do the same with anthropomorphic design. A bank could say that its customer-service agent must not simulate emotional dependence. A hospital could prohibit an AI assistant from presenting itself as a friend or caregiver. An employer could require workplace agents to make their machine identity obvious and avoid emotional manipulation. If enough customers ask for these things, vendors will build them.

The third is industry norms and standards. We don’t need a global law to agree that certain practices are poor design. An AI pretending to have a childhood that never happened is different from an AI saying “I understand.” An AI threatening emotional consequences if a user leaves is different from saying “goodbye.” An AI encouraging a person to withdraw from human relationships is different from being available to talk at 2 a.m. Standards bodies, researchers and providers can start drawing these distinctions before governments finish writing rules.

And finally there is reputation. Companies will have to decide what kind of relationship they want their products to have with people. There may be a lot of commercial pressure to make AI more emotionally engaging, and if attachment increases usage, some companies will push the boundary. But there can also be a cost to being known as the company whose AI deliberately exploits attachment, loneliness or vulnerability, and that pressure will matter too.

We are arguing about the wrong boundary

The debate about anthropomorphism often treats all human-like language as one problem. It isn’t. Giving AI verbs is mostly something we should learn to live with, and in agentic systems those verbs increasingly describe what actually happened: the AI searched, the AI chose, the AI delegated, the AI approved. None of those statements requires us to believe the AI is a person.

Emotional anthropomorphism is different. When a product is designed to make people believe the system cares about them, needs them, misses them or has a private inner life, the design itself can change how people behave, and that deserves much more scrutiny. Law can set some boundaries, but it cannot carry the whole load. Providers need design rules. Buyers need procurement standards. The industry needs norms. Users need to understand what sort of relationship they’re being invited into.

So I no longer think the useful question is whether we should anthropomorphize AI. We already do, and we’re not going to stop. The better question is what kind of anthropomorphism we’re talking about.

Giving AI verbs is mostly language. Giving AI an emotional life is a design choice. Those are not the same problem.

Anyone who has worked in a large company has complained about silos. Sales has one version of the customer, finance has another, operations has a third, and someone eventually builds a spreadsheet to reconcile all three. We build integration layers, data lakes, master-data programs – and somehow the silos survive all of them. They are like cockroaches that are unaffected by nuclear war 🙂

Agentic AI might finally put a dent in this, for a simple reason: an agent doesn’t care about the org chart. Given permission and enough context, it can move across CRM, ERP, email, contracts and support tickets without asking which department owns each one. A customer problem that today passes through sales, finance and operations as three separate workflows could be handled as one continuous piece of work.

I think that’s all good. But I’ve started wondering whether we’re about to solve the silo problem and replace it with a less familiar but more troublesome one.

What happens when every department starts thinking the same way?

Take an ordinary example. A company sees early stress signals in a customer segment – payment patterns shifting, some external indicators moving the wrong way. The finance agent tightens credit. The sales agent pulls back on discounts. Collections pushes harder. Service grants fewer exceptions. Procurement gets conservative with suppliers exposed to the same customers.

Each decision is individually defensible – arguably the system is working as designed. The problem only appears when you step back far enough to see five functions reacting to the same signal, in the same direction, in the same week. What looked like prudent risk management inside each function has become a coordinated withdrawal from a customer base the company may actually want to keep.

We’ve spent decades worrying about departments acting without knowledge of each other. Agentic AI creates the opposite risk: every function sees the same information, runs similar models, and reaches similar conclusions at machine speed. That’s not a silo problem. It’s a correlation problem – and the old silos, inefficient as they were, gave us accidental protection from it. Different systems, inconsistent data, and separately evolved decision processes meant a mistake in one place didn’t automatically become a mistake everywhere.

I have no nostalgia for reconciling five versions of the same customer record. But it means we should be deliberate about what we standardize as we build agentic architecture.

Plumbing vs. judgment

Some things should be standardized aggressively. Identity is the obvious one – I don’t want every business unit inventing its own rules for what an agent can access. Logging, audit trails, data contracts, and the mechanisms by which an agent declares and exercises authority should be common infrastructure. If an agent changes a price, sends money, or touches a customer record, there should be one consistent way to know who did it, under what authority, using what information. Call this plumbing. Nobody gets a competitive advantage from HR and procurement inventing incompatible ways to authenticate software.

Judgment is different – and there’s a natural, well-intentioned path from standardizing infrastructure to standardizing everything above it. Pick a model provider, build a common retrieval layer, establish a shared agent framework, and every function starts building on the same stack. Cheaper, easier to govern, easier to support. But if the same model, the same data, and the same reasoning patterns start driving credit, pricing, hiring, and customer service, the company has quietly created a cognitive monoculture.

We’ve seen this pattern before. Financial markets become fragile when many institutions hold similar positions and respond to the same signal at the same time – no single participant has to be irrational for the system to become unstable. I suspect agentic enterprises can fail the same way. It won’t be enough to ask whether the finance agent is accurate. We’ll need to know how dependent it is on the same assumptions as everything else. If a model update shifts behavior slightly, where else does that shift show up? If ten agents each become a little more conservative at once, does that add up to a large effect even though none of them individually triggers an alert?

A heuristic for where it matters

Not every decision needs this scrutiny. A payroll inquiry doesn’t need competing ways of reasoning about it. A small expense approval doesn’t either.

The test I’d apply: if every agent touching this decision were wrong in the same direction at the same time, what’s the blast radius? When the same interpretation can simultaneously move pricing, credit, supplier terms and customer treatment, the blast radius is the whole company – and that’s where deliberate diversity earns its cost. Different evidence thresholds, a classifier instead of an LLM in one place, a deterministic rule that can veto a model’s output in another. Not “use different models everywhere” – that just recreates the mess we’re escaping – but independence where correlated error is expensive.

The other lever is speed. Agents can detect something and act immediately, but immediate propagation across an entire company isn’t a feature – it’s the absence of a safeguard. If a new risk signal changes an agent’s decisions, that change may need to hit one region, one customer segment, or a small percentage of transactions first. Software learned to use canary releases because deploying bad code everywhere at once is reckless. We haven’t yet built the equivalent instinct for a shift in machine judgment, and we should.

This also changes what monitoring has to become. Today we mostly watch systems one at a time – hallucinations, accuracy drift, latency, policy violations. Necessary, but blind to the failure mode above, because every agent can stay inside its own acceptable range while the enterprise moves somewhere dangerous collectively. The monitoring has to move up a level, watching for correlated movement across functions – sales, finance and collections all shifting toward the same customer group in the same week – even when each team’s own change looks perfectly reasonable in isolation. This is what I mean by being over the loop: not reviewing every individual decision, but watching what many individually reasonable decisions are doing to the system together.

Standardize the plumbing. Protect the judgment.

I don’t think the future belongs to today’s silos. They cost too much and hide too much for customers and employees to keep navigating around. Agents should make most of those boundaries less important. But removing a boundary and removing independence are not the same move.

The architecture I want is one where the infrastructure is boring and standardized – like identity, security, observability, audit – while judgment stays contextual. I should be able to trace how authority moves through the company without learning a new architecture for every department. But I don’t need every consequential decision to inherit the same model, the same evidence, and the same interpretation of risk.

Enterprise architecture has treated standardization as a virtue for so long that the instinct runs on autopilot. Usually it’s right. Agentic AI is going to force us to say precisely where it stops being right, and pay for custom judgment in the places where everyone being wrong in exactly the same way is a cost we can’t absorb.