Essay
·Enterprise·6 min read
Every enterprise deploying LLM features ends up building an eval harness, a trace store, and a guardrail layer. Most build them badly, twice. Where the ownership boundary should actually sit.
By Dr Gareth Roberts
There is a layer of infrastructure that almost every enterprise deploying LLM features ends up building, usually without deciding to. It starts as a spreadsheet of test prompts. Then a script that runs them. Then a dashboard someone built in a sprint. Eighteen months later there is an internal platform with three part-time maintainers, no documentation, and a backlog nobody is funded to clear — and the second product team, discovering it does not fit their use case, has started building their own.
This layer — evaluation, tracing, guardrails, release gating for model-driven behaviour — is the control plane for production AI. The question of whether to build or buy it deserves more deliberation than it usually gets, because the default path is not “build”. The default path is “accrete”, and accretion produces the worst of both options: the ongoing cost of ownership without the design that makes ownership worthwhile.
Three properties make the control plane unusually prone to bad internal builds.
It looks trivial at every individual step. Running a set of prompts against a model and diffing the outputs is an afternoon of work. Logging requests and responses is a logging statement. Each increment is small enough that no one writes a design document, requests headcount, or evaluates alternatives. The system that results was never designed; it was deposited.
It sits outside every team’s mandate. The product team is measured on features. The platform team is measured on infrastructure the whole company uses, and one team’s eval scripts do not qualify until it is too late to consolidate. The ML team, if there is one, is measured on model quality, not on the operational tooling around it. The control plane is everyone’s dependency and no one’s deliverable.
Its real requirements only surface in production. The prototype needs to answer “did the output change?”. The production system needs to answer “which of the forty prompt variants currently live is responsible for the regression a customer reported on Tuesday, under which model version, and what does the trace show?”. Teams discover requirements such as versioned datasets, statistical comparison across non-deterministic runs, trace retention policies, and per-tenant isolation one incident at a time, and retrofit each into a codebase that assumed none of them.
The result is familiar from other infrastructure categories. Most companies no longer write their own metrics pipelines, CI runners, or feature-flag systems, and the reasons they stopped apply with at least equal force here. The difference is that those categories had a decade to mature; this one is being rebuilt in parallel, right now, inside thousands of organisations that each believe their requirements are unusual.
Buy-vs-build analyses usually compare licence fees against the initial engineering effort, which is the one comparison that flatters building. The initial effort is genuinely small. The costs that dominate arrive later and rarely appear on anyone’s ledger.
Maintenance against a moving substrate. Model APIs change: new providers, new parameters, new modalities, deprecations on a provider’s schedule rather than yours. An internal tool built against one vendor’s completion API needs continuous work merely to stand still. Vendors amortise this across their customer base; you carry it alone.
The evaluation science, not just the plumbing. Storing eval results is easy. Knowing that a judge model needs its own calibration, that pass-rate deltas on a fifty-case suite are mostly noise, that position bias corrupts pairwise comparisons — that is accumulated, specialised knowledge. Internal tools tend to get the plumbing right and the statistics wrong, which is worse than having no tool, because it produces confident, wrong release decisions.
The bus factor. These tools are typically maintained by one or two people who built them out of necessity. When they move on, the organisation owns a system it cannot safely change, sitting in the release path of every AI feature.
Opportunity cost. The strongest engineers on an AI product team are exactly the people who end up maintaining the internal harness, and exactly the people whose time on it is most expensive.
A reasonable rule of thumb: whatever the initial build cost, ownership over three years costs a multiple of it, and the multiple grows with the number of teams the tool must serve.
The right boundary is not “build everything” or “buy everything”. It follows from a simple criterion: own what encodes your judgement; standardise the machinery that executes it.
Own your evaluation content. Your eval datasets, grading rubrics, and pass thresholds encode what your organisation means by “good” for its specific products and obligations. No vendor can supply these, and any vendor who claims their generic benchmarks substitute for them is selling you a false sense of coverage. This is also the asset with lasting value: datasets built from your real failure cases survive every change of model, vendor, and framework.
Own your policy decisions. Which actions an agent may take autonomously, what requires human sign-off, what your guardrails must block, what your risk function will accept — these are decisions, not infrastructure, and they must remain legible to your own compliance and security functions.
Standardise the execution machinery. Runners, trace collection and storage, comparison statistics, dashboards, alerting, dataset versioning, judge-model orchestration. This is undifferentiated by definition: nothing about your business changes how a trace should be stored or how a paired comparison should be computed. It is also where the hidden costs above concentrate.
Standardise the interfaces even where you build. If you do build components — and there are legitimate reasons, covered below — build against open formats. Emit traces as OpenTelemetry spans rather than a bespoke schema. Keep eval datasets in portable formats with version control rather than inside a tool’s database. The largest avoidable cost in this category is not building the wrong thing; it is coupling your judgement (which you must keep) to your machinery (which you should be able to replace).
The honest cases for building are narrower than most teams believe, but they exist.
What does not qualify: “our requirements are unique” (they are usually unique in content, which you own anyway, not in machinery), “we already started” (sunk cost), and “an engineer built most of it in a week” (see the multiple above).
Before committing to an internal build, an engineering leader should be able to answer all of the following in writing:
If the answers are vague, the organisation is not choosing to build. It is choosing to accrete, and it will make this decision again, under worse conditions, after the first serious incident.
Share
More
