Priors over priors

People ask about the name. The honest answer takes a short detour through Bayesian statistics, and the detour turns out to be the company’s thesis, so it is worth writing down properly.

What a hyperprior is

In Bayesian modelling, a prior is what you believe before you see the data. Observe evidence, apply Bayes’ rule, and the prior becomes a posterior — belief updated in proportion to what the evidence actually supports.

But priors themselves have parameters, and a modeller must ask where those come from. The Bayesian answer is to go one level up: place a distribution over the parameters of the prior. That distribution is a hyperprior — a prior over priors. It encodes beliefs not about the world directly, but about what kinds of beliefs are plausible in the first place: how variable things tend to be, how strongly any single observation should move you, how much structure to expect before the data arrives.

The practical effect is a kind of principled humility. Hierarchical models with well-chosen hyperpriors resist overreaction. A handful of surprising observations moves the posterior a little, not catastrophically, because the model holds beliefs about how quickly beliefs should change. Estimates for sparse groups borrow strength from the population instead of chasing noise. The hyperprior is the level of the model that governs the temperament of every level below it.

That is the concept we named the company after, and not because it sounds pleasingly technical. It is a precise description of the layer we build.

Models hold beliefs; systems must hold beliefs about them

A language model is, functionally, an enormous learned prior over text and action — compressed belief about what follows from what. In production, you condition that prior on a context window and sample from the posterior. This is a serviceable cartoon of what happens on every request.

The cartoon exposes the operational question, which is not “what does the model believe?” but “how should the system treat what the model believes?” A model will produce an answer with the same fluent confidence whether it is on well-trodden ground or extrapolating cheerfully into the void. The weights do not know which deployment they are in, what the cost of a wrong answer is, or that the tool schema changed on Tuesday. Something above the model has to hold that knowledge.

That something is a second level of belief — belief about the model’s beliefs. When is this model trustworthy, and on what distribution of inputs? How much should one confident output move a consequential decision? What evidence would indicate the model’s competence has shifted? These are hyperprior questions, and every element of a serious control plane is an answer to one of them:

• Evaluation is how the system learns what the model is actually good at — a posterior over competence, grounded in evidence rather than vibes, updated every time the suite runs.
• Guardrails encode beliefs about where model output should not be deferred to at all: regions of behaviour where no amount of fluency constitutes evidence, and policy overrides the prior outright.
• Observability is the evidence stream. Traces, drift signals, cost and latency telemetry — the data against which beliefs about the model are continuously updated. Drift detection is, quite literally, noticing that the world has moved away from your prior.
• The harness is where updated belief becomes action: route to this model, retry that call, lower the autonomy, escalate to a human. Deference, made mechanical.

Calibration, or the virtue that matters most

Of all the properties you could want from a production AI system, calibration is the one we would defend to the last: when the system acts as if it is 90% sure, it should be right about nine times in ten.

Note that this is a property of the system, not the model. Raw model confidence — logprobs, verbalised certainty, tone — is famously unreliable, and fluency is not evidence. Calibration has to be manufactured at the level above: measured on eval sets, tracked per task and per slice, corrected where the model is systematically over- or under-confident, and — critically — wired to consequences. A calibrated confidence estimate that does not change what the system does is decoration.

This is where the Bayesian framing pays rent. A well-specified hierarchical model is calibrated because of its hyperpriors: the upper level keeps the lower level from claiming more certainty than its evidence supports. The control plane plays the same role for a deployed model. It is the level of the system whose job is to keep every level below it honest about what it knows.

Deference is a dial

The deepest design question in AI operations is not “what can the model do?” but “how much should we defer to it, here, today?” — and the answer should never be a constant.

Deference should vary with the stakes: high autonomy for drafting, low for irreversible actions. It should vary with the evidence: a model that has earned a strong track record on a task, demonstrated through evals and production history, warrants more room than one that has not. And it should vary over time, because models drift, inputs drift, and last quarter’s posterior is this quarter’s prior — due for an update, not for retirement into dogma.

Systems without an explicit deference dial still have one; it is simply set once, implicitly, by whoever wrote the first integration, and never revisited. Most AI incidents we see are miscalibrated deference wearing a technical costume: the system kept trusting after the evidence turned, or never established grounds for trust before granting it.

Beliefs about how to hold beliefs

There is a modelling folk theorem worth stealing: you cannot escape assumptions, you can only choose whether to state them. A flat prior is still a prior. An AI deployment with no explicit control plane still has one — its beliefs about model trust are merely unexamined, scattered through retry logic and half-remembered Slack threads.

We named the company Hyperpriors because the alternative to an explicit layer of beliefs-about-beliefs is an implicit one, and implicit ones fail quietly. The work is to make that layer deliberate: evidence in, calibrated trust out, deference dialled to what the model has actually demonstrated. A prior over priors, running in production.

And yes — we are aware that naming the company after a regularisation strategy is the most on-brand form of restraint available to us.