Defence in depth for LLM applications

Every few months a team discovers that their carefully worded system prompt did not, in fact, prevent the model from doing the thing the prompt forbade. The discovery is usually followed by a better prompt, which fails less often, and then a moderation filter, which fails differently, and eventually — sometimes after an incident — the realisation that they have been building a layered safety architecture all along, just reactively and in the wrong order.

This paper makes the case for building it deliberately. The organising principle is old: defence in depth, the assumption that every individual control is leaky and that safety comes from stacking controls whose leaks do not line up. What is specific to LLM applications is which layers you need, where each one leaks, and the engineering required to keep their failures independent — because independence is the entire value of the design, and it is easy to build five layers that fail as one.

The architecture has five layers. In order, from the outside in: input mediation, capability scoping, output policy enforcement, human review, and audit. Each is described below with its job, its characteristic failure, and its relationship to the others.

Layer 1: Input mediation

Everything that enters the model’s context is either an instruction you wrote or data you did not. The first layer’s job is to keep that distinction legible — to the system, since the model itself cannot be trusted to maintain it.

Concretely, input mediation covers:

• Structural validation. Size limits, format checks, encoding normalisation, stripping of control characters and invisible Unicode. Mundane, and the precondition for everything downstream: a filter cannot inspect what it cannot parse.
• Provenance labelling. User messages, retrieved documents, tool results, and third-party content each enter the context wrapped in structure that records where they came from. The application should be able to answer “which input influenced this output” without forensics.
• Injection surface reduction. Any content the operator did not write — a fetched web page, an email being summarised, a PDF from a customer — is a potential carrier of adversarial instructions. Mediation cannot reliably detect injection (nothing can, yet), but it can quarantine: delimiting untrusted content clearly, refusing to let it introduce new instructions by policy, and flagging inputs containing instruction-shaped text for closer handling.
• Tenant and session isolation. One user’s documents, history, and retrieved context must be constructible into another user’s window only by explicit design, never by cache accident or shared retrieval index misconfiguration. Cross-tenant leakage through the context window is an underreported class of incident because it looks like a retrieval bug rather than a security failure.

Where it leaks. Input mediation is a classifier problem over adversarial, natural-language input; it will miss things. A well-crafted injection reads exactly like benign content. The layer’s realistic goal is to stop the casual and the accidental, raise the cost of the deliberate, and label everything so that later layers know what they are looking at.

Layer 2: Capability scoping

If input mediation limits what the model hears, capability scoping limits what the model can do. This is the layer with the best return on engineering effort, because it is the one place where controls are deterministic: a credential the process does not hold cannot be exfiltrated, and an API the tool cannot reach cannot be misused.

The design rule is that the model’s tool list is a vocabulary, not a permission set. Authority lives in the execution layer, and it is scoped along four axes:

• Which tools this agent, in this deployment, may invoke at all. A summarisation feature has no business holding a send_email tool because the platform happens to offer one.
• Which arguments are acceptable. Not just schema validity but domain constraints: refund amounts capped, file paths jailed to a working directory, SQL restricted to read-only replicas, outbound requests restricted to an allowlist of domains. Argument constraints are where most real incidents are actually stopped.
• Which identity the call executes under. Tools should run with per-invocation, least-privilege credentials scoped to the current user and task — never a god-mode service account. When an agent acts on behalf of a user, it should be able to do at most what that user could do, and usually less.
• What consequence class the call belongs to. Reads, reversible writes, irreversible writes, and external communications are different categories, and the harness should know each tool’s class, because layers 4 and 5 key off it.

Sandboxing belongs here too: code execution in disposable containers, network egress denied by default, filesystem access scoped and ephemeral. The question to keep asking is the blast-radius question — if the model made the worst plausible sequence of calls this layer permits, what is the damage? If the answer is unacceptable, the fix is in this layer, not in the prompt.

Where it leaks. Scoping fails through composition. Each tool is individually safe; a sequence is not — a read tool that ingests attacker-controlled content followed by a send tool with a permissive argument policy is an exfiltration pipeline assembled from two “safe” parts. Reviewing tools one at a time misses this; you have to review the reachable combinations.

Layer 3: Output policy enforcement

Between the model producing an output and that output reaching a user or a system sits the third layer: enforcement of what the product is allowed to say and emit, checked mechanically on the way out.

This layer has two distinct kinds of check, and conflating them causes trouble:

• Deterministic checks — schema validation for structured output, redaction of credential patterns and personal data formats, allowlists for links and phone numbers the product may present, formatting and length constraints. These are cheap, fast, and testable like ordinary code. Every output should pass through them.
• Statistical checks — model-based classifiers for policy categories (harmful content, regulated advice, off-brand claims), and groundedness checks that verify a generated answer is actually supported by its cited sources. These are more capable and less reliable, and they cost latency and money, which in practice means applying them by risk tier rather than uniformly.

Two design decisions matter more than the checks themselves. First, fail closed: when an output check errors or times out, the safe default is to withhold or route to review, not to ship the unchecked output. This must be an explicit decision because the convenient default in most codebases is the opposite. Second, define what happens on rejection: silently retrying the model with the violation appended is often fine; doing so more than twice usually means the case should escalate rather than iterate.

Where it leaks. The statistical checks share a weakness with the model they police: they are pattern-matchers over language, and content designed or unlucky enough to sit off their training distribution passes through. The deterministic checks leak differently — they only catch what someone anticipated. The pairing is the point: one catches the unanticipated-but-typical, the other the anticipated-and-precise.

Layer 4: Human review loops

The fourth layer inserts people, and the engineering challenge is inserting them where they change outcomes rather than where they merely add latency. Human review is a scarce, expensive, fatigable resource; the architecture’s job is to spend it precisely.

Three placements cover most products:

• Pre-execution approval for actions in the irreversible or external-communication consequence classes — the classes defined in layer 2. The system pauses with a proposed action, the evidence for it, and a specific question. This is the control of last resort before the world changes, and it is only viable if the volume is kept low enough that reviewers genuinely review.
• Uncertainty-routed escalation for cases the system identifies as likely wrong — low verifier scores, retry loops, contradictions with retrieved evidence. The calibration of these routes is its own discipline, covered at length in our piece on uncertainty routing; the architectural point is that the route must exist as a designed workflow with state, SLOs, and resolution paths, not as an exception handler.
• Post-hoc sampled audit for everything else. A random sample of automated outputs, reviewed against ground truth, is what tells you the automated path still deserves its autonomy. It is also the supply line for evaluation data.

The known failure of this layer is rubber-stamping: flood reviewers with escalations that did not need them and approval becomes reflexive, at which point the layer is theatre. Queue precision — the fraction of escalations where the human actually changed the outcome — is the health metric, and it should be on a dashboard next to the error rates.

Where it leaks. Humans miss things, tire, anchor on the model’s proposal, and inherit the model’s framing of the case. Review catches a different distribution of errors than the automated layers — obvious-to-a-person absurdities, contextual inappropriateness, judgement calls — and misses some things a regex would catch. That is acceptable. It is one layer, not the layer.

Layer 5: Audit trails

The final layer prevents nothing in the moment, and is for that reason the easiest to deprioritise and the most regretted omission after an incident. The audit trail is the record of every boundary crossing in the previous four layers: what entered the context and from where, what the model proposed, what the mediation and scoping layers decided and under which policy version, what the output checks scored, who reviewed what and what they decided.

The requirements that distinguish an audit trail from ordinary logging:

• Completeness at the boundaries. Every tool call with full arguments and results; every policy decision with the rule and version that made it; every context assembly with the provenance of its parts. If the question “why did the system do this” cannot be answered from the trail alone, the trail is incomplete.
• Integrity. Append-only storage, separated from the application’s write path, with retention aligned to your regulatory reality. A trail the application can rewrite is a narrative, not a record.
• Correlation. One identifier that threads a request through every layer, model call, tool invocation, and review decision. Post-incident reconstruction time is roughly proportional to how well this was done.
• Replayability. Enough recorded state to re-run an interaction against a new model, policy, or check. This is what turns incidents into regression tests and makes the audit trail a feeder for the evaluation suite rather than a write-only archive.

Where it leaks. Trails fail by omission (the one unlogged code path is where the incident happened), by unqueryability (the data exists but no one can extract the answer under incident pressure), and by becoming a liability themselves — an audit trail full of unredacted personal data is a second breach waiting inside the first. Treat the trail as a production system with its own access controls and minimisation policy.

Independence is the load-bearing property

Stacking five leaky layers only helps if their leaks are uncorrelated. This is the part of defence in depth that decays silently, because correlations creep in through convenience:

• Shared models. Using the same base model as generator, injection detector, output classifier, and verifier means one set of blind spots wearing four uniforms. An input that fools the generator has a materially elevated chance of fooling its siblings. Diversify where it matters: different models, or at minimum genuinely different prompts and vantage points, for checking versus generating.
• Shared code paths. A parsing bug in a common library, a serialisation quirk, one feature-flag service — any of these can disable multiple layers simultaneously. The layers should fail independently in the mundane operational sense too: an outage in the output classifier should degrade to fail-closed behaviour, not take the audit trail down with it.
• Shared assumptions. If every layer assumes tool results are well-formed, or that user IDs in the context are authentic, then the assumption itself is a single point of failure sitting underneath the entire stack. Assumptions each layer relies on should be written down and, where possible, enforced by a different layer.

The test worth running periodically: take each layer, assume it silently fails open, and trace what the remaining layers catch. If any single-layer failure yields an unacceptable outcome, that is not a five-layer architecture; it is one control with four decorations.

There is a converse signal, too. A layer that never fires in production is either redundant or broken, and monitoring should distinguish the two. Healthy defence in depth shows each layer catching a modest, boring stream of real events — that stream is the evidence the layer exists in practice and not just in the architecture diagram.

Build order

For teams starting from a prompt and a hope, sequence matters, because the layers differ sharply in cost and coverage:

1. Capability scoping first. It is deterministic, it bounds the blast radius of everything else, and it works even on the day your classifiers are wrong. Least-privilege credentials and argument constraints are the highest-value week of safety engineering available.
2. Audit second. Before more controls, visibility — you cannot tune layers you cannot observe, and every subsequent layer becomes cheaper to validate once the trail exists.
3. Deterministic output checks third, then input mediation, then the statistical checkers as evaluation data accumulates to calibrate them.
4. Human review last as a formalised layer — not because people matter least, but because a review loop built before the routing signals and queues exist degenerates into the rubber-stamp pattern immediately.

None of these layers will hold on its own. Each will fail in production, observably, and if the architecture is sound that will be an ordinary Tuesday rather than an incident: one layer’s miss, another layer’s catch, a line in the audit trail, a new case in the evaluation suite. That is what safety looks like in systems built from statistical components — not the absence of failure, but the refusal to let any single failure be enough.