Uncertainty routing in production

The question that decides whether an LLM system can be deployed on consequential work is rarely “is the model accurate enough”. It is “does the system know when it is likely to be wrong, and what does it do about it”. A model that is right 94% of the time and silent about the rest is a liability; the same model, routing most of that 6% to a person, is a workforce multiplier.

That routing decision — automate, or escalate — is a piece of engineering in its own right. It has inputs (confidence signals), a policy (thresholds), a destination (a human with a queue), and a failure mode all of its own (alert fatigue). Teams that treat it as an afterthought end up in one of two ditches: automating errors, or drowning reviewers until they stop looking.

Confidence signals and their limits

There is no single number that tells you a model is uncertain. There is a family of signals, each cheap or expensive, each with a characteristic way of lying to you.

Token-level probabilities. Where the API exposes them, logprobs over the answer tokens are the cheapest signal available. They are useful for narrow, constrained outputs — a classification label, a yes/no gate — and progressively less meaningful as outputs get longer and more open-ended, where high per-token confidence can coexist with a confidently wrong answer. They are also model-specific: swap the model and every threshold you calibrated is void.

Verbalised confidence. Asking the model to rate its own certainty is popular because it is easy. Treat it as weak evidence: self-reported confidence is poorly calibrated out of the box, sensitive to prompt phrasing, and prone to the same failure as the answer itself — a model that has misread the question will misreport its confidence with equal fluency. It earns a place only after you have measured its calibration on your own traffic.

Verifier models. A second model (or the same model with a checking prompt) that grades the first model’s output against the source material. More expensive, but it changes vantage point: the verifier sees the answer as an artefact to inspect rather than a sequence to continue, and that catches a different class of error. The limit is correlation — two calls to the same base model share blind spots, so a verifier is a real second opinion only to the degree that its prompt, inputs, or model differ.

Agreement signals. Sample the answer more than once and measure consistency; or, in retrieval systems, measure whether the answer is actually entailed by the retrieved passages. Disagreement between samples, or an answer that floats free of its citations, is one of the more reliable uncertainty indicators available — and one of the more expensive.

Behavioural signals. These are free and widely ignored: the model needed three attempts to produce parseable output; a tool call failed twice; the conversation has looped; the user has rephrased the same request a third time. None of these are confidence scores, but all of them are evidence that this interaction is off the happy path, and routing policies should consume them.

The practical pattern is to combine two or three signals with different failure modes — typically one cheap intrinsic signal, one behavioural signal, and one verifier on the traffic that matters — and to accept that the combination is a heuristic to be validated empirically, not a probability you can reason about from the armchair.

Calibrating the threshold

A threshold is a business decision wearing a statistical costume. Setting it requires exactly one thing most teams skip: a labelled sample of production-representative traffic, scored by the same signals the router will see, with ground truth about which outputs were acceptable.

With that in hand, the procedure is unglamorous:

1. Plot the trade-off. For each candidate threshold, compute the automation rate and the error rate among automated cases. This is a curve, and someone accountable for the product should look at it.
2. Price both errors. A false escalation costs one unit of reviewer time. A false automation costs whatever a wrong answer costs in your domain — an annoyed customer, an incorrect refund, a compliance report. The asymmetry between those two prices, not a round number like 0.8, determines the operating point.
3. Set thresholds per intent, not globally. “Reset my password” and “close my account and transfer the balance” do not deserve the same tolerance. A single global threshold is a sign the routing layer was designed in a hurry.
4. Recalibrate on every change. New model version, new prompt, new retrieval index, or measurable drift in the traffic mix — each of these silently moves the meaning of your scores. Escalation rate itself is the cheapest drift alarm available: alert on it in both directions, because a falling rate may mean the model has become confidently worse.

A routing policy is worth writing down as configuration rather than burying in code, because the people who need to review it are not always the people who wrote it:

route: refunds
signals: [verifier_score, retry_count, amount]
rules:
  - if: amount > 250
    then: escalate            # policy, regardless of confidence
  - if: verifier_score < 0.72 or retry_count >= 2
    then: escalate
  - else: automate
review_slo: 15m

Note the first rule: some escalations are not about uncertainty at all. High-consequence actions go to a person even when every signal is green, because confidence is an estimate and some mistakes are not priced in estimates.

Escalation ergonomics

Where escalations land determines whether the mechanism works. The commonest failure is handing the reviewer a raw transcript and a vague instruction to “check this” — converting a two-minute decision into a twenty-minute reconstruction, to which reviewers respond the way any rational person does: they skim.

A well-designed escalation is a case file, assembled by the system:

• The specific question. Not “review this conversation” but “the customer requests a £310 refund; the order history supports £185; approve, adjust, or decline”. If the system cannot state what it needs the human to decide, it has escalated too early or too vaguely.
• The proposed action and the evidence. What the model wanted to do, and the passages, records, or tool results it based that on — with the contradicting evidence surfaced, since that is usually why the case escalated.
• One-click resolution paths. Approve, edit-and-approve, decline with reason. Every resolution is labelled data; a review queue that does not feed the evaluation suite is wasting its most valuable output.
• Clean pause and resume. The automated flow should hold state, inform the end user honestly about the delay, and resume from the human’s decision without replaying or re-deciding anything.

Latency is part of ergonomics. An escalation path with a four-hour median response time will be routed around — by product managers who quietly raise thresholds, and by users who abandon the channel. If you cannot staff the queue to its SLO, you have a place where cases go to age, not an escalation path.

The alert-fatigue budget

Every escalation spends reviewer attention, and attention is a depleting resource with a sharp failure mode. When the queue fills with cases the model would have handled correctly, reviewers learn — accurately — that most alerts are noise. Approval becomes reflexive. At that point you have the cost of human review and the safety of none, which is the worst configuration available.

Treat the queue’s precision as a first-class metric: of the cases escalated, what fraction actually needed a human to change the outcome? If that number sits below something like one in five, the routing layer is training its reviewers to rubber-stamp. The corrective levers, in rough order of preference:

• Raise the threshold on the intents generating the noise, using the calibration data to bound the added risk.
• Deduplicate. Ten escalations with the same root cause are one incident, not ten decisions.
• Tier the queue: quick-glance approvals separated from genuine judgement calls, so the expensive attention goes where it is needed.
• Sample the automated side. A small random audit of auto-approved cases keeps you honest about what the raised threshold is letting through.

The steady state to aim for is a queue where most items genuinely need a person, resolved within its SLO, whose outcomes flow back into evaluation and threshold calibration. Get there and escalation stops being the embarrassing exit from automation. It is the control loop that lets you automate more next quarter than you safely could this one.