Discussion of agent safety tends to concentrate on whether the model will make a mistake. That is the wrong variable to design around, because the answer is settled. It will.
The variable worth engineering is what a mistake can reach. An agent that misjudges a task and edits one file in a scratch directory is a minor annoyance. An agent that misjudges the same task while holding production database credentials is an incident. The model behaved identically in both cases.
Capability and authority are separate
Two properties are routinely conflated. Capability is what the model is competent to do. Authority is what the surrounding system permits it to do.
Improving capability is slow, uncertain, and largely outside your control. Restricting authority is immediate, reliable, and entirely within it. A system whose safety depends on the model reasoning correctly has been built on the property that is hardest to guarantee.
Stated as a design rule: assume every action the agent can take will eventually be taken at the wrong moment, on the wrong target, or repeatedly. Then decide whether the system survives that.
How agents actually fail
The failures fall into recognisable categories, and each has a different containment.
The wrong action, where the agent selects a tool that does not fit the task. The right action on the wrong target, which is the more common and more damaging case, because the action itself looks entirely reasonable in the log. The repeated action, where an agent that receives an error retries unchanged, which turns a single mistake into a sustained one. And the cascading action, where an early wrong step produces state that makes every subsequent step wrong while remaining internally consistent.
The last is the hardest to detect, because nothing in the trace looks anomalous. Each step follows sensibly from the one before, and the whole sequence is wrong.
Reversibility is the axis that matters
If one property is worth designing around, it is whether an action can be undone. Reversible actions are cheap to get wrong, because the recovery path is short. Irreversible ones are not, regardless of how confident the agent was.
The useful practice is to sort every available tool onto that axis explicitly and treat the two groups differently. Reading is reversible. Writing to a scratch location is effectively reversible. Sending a message, deleting a record, transferring money, or changing a production configuration is not.
Irreversible actions deserve a gate that does not depend on the model's judgement, which usually means an explicit confirmation from a person, or a narrow allow list of exactly which targets are permitted. Placing that gate in the runtime rather than the prompt matters, because an instruction is a preference while a check is a property.
Sorting the tools by reversibility
| Action | Reversible | Appropriate gate |
|---|---|---|
| Reading data | Yes | Scope of credentials |
| Writing to a scratch location | Effectively | Step and spend limits |
| Modifying production state | No | Explicit confirmation |
| Sending a message externally | No | Allow list of destinations |
| Deleting or transferring | No | Explicit confirmation |
Sorting every available tool onto that axis before deployment takes minutes and determines which failures are recoverable.
Containment in practice
The techniques are ordinary and mostly predate agents entirely.
Give the agent the narrowest credentials that permit its purpose, and prefer read only wherever the task allows it. Separate environments so that exploratory work cannot touch production. Offer a mode that reports what would be done without doing it, which is the cheapest possible safety measure and is frequently omitted. Cap the number of steps, the total spend, and the rate of actions, so that a loop which has gone wrong stops on its own rather than continuing until someone notices.
Keep a complete trace. When something goes wrong the useful artefact is the full sequence of state, decision, and result, because the originating error is nearly always several steps upstream of where the damage appeared. A system without that record is diagnosed by reconstruction and guesswork.
Untrusted content makes this an attack surface
Everything above assumes mistakes are accidental. Once an agent reads content it did not author, they need not be.
Anything a tool returns enters the context as text, and a model does not reliably distinguish data it retrieved from instructions it was given. A web page, a document, an issue description, or an email can therefore contain text intended to steer the agent, and the agent will act on it using its own credentials.
The confused deputy, again
This is the classical arrangement in which a privileged component acts on instructions from an untrusted source, and the defence has always been the same. Do not rely on the deputy recognising the manipulation. Constrain what the deputy is able to do, so that following a hostile instruction produces a small effect rather than a large one.
Practically, that means an agent which reads external content should hold less authority than one operating only on data you control, and it means the boundary between the two should be a deliberate design decision rather than an accident of which tools happened to be connected.
Detection time is part of the radius
Blast radius is usually discussed as scope, meaning what the agent can touch. Duration belongs in the same calculation, because damage is scope multiplied by how long it continues before anyone notices.
An agent acting wrongly for thirty seconds before a check stops it is a different event from one acting wrongly overnight. This is the argument for limits that trigger automatically rather than alerts that require someone to read them, and for treating an unusually high action rate as a reason to halt rather than a metric to graph.
It is also the honest limit of human approval as a safety mechanism. Requiring confirmation on every action does contain the radius, and it also removes the reason to use an agent, so in practice approval gets granted quickly and with declining attention. Approval works when it is reserved for the small set of irreversible actions and the reviewer therefore sees few enough of them to consider each one. Applied to everything, it degrades into a button people press.
Several agents multiply the problem
Systems that decompose work across multiple agents inherit every issue above and add two more.
The authority of the system becomes the union of every agent's permissions, which is usually larger than anyone intended and is rarely written down anywhere. And the output of one agent becomes the input of another, so a wrong conclusion is laundered into apparent fact by the act of being passed along, arriving at the next agent with no indication that it was uncertain.
The mitigation is unglamorous. Keep the permission set of each agent explicit and minimal, and pass structured results with their provenance rather than prose that has lost track of where its claims came from.
Note: the design question worth asking before deployment is not how often the agent will be right. It is what the worst thing it can do is, how quickly that would be noticed, and how long it would take to undo.