Evaluation-Driven Development Rests on Judges Whose Reliability Is Rarely Measured

Dear Reader…

The pitch for evaluation-driven development is easy to agree with, which is part of the problem. Stop shipping LLM features on the strength of a demo and a product manager's intuition. Write the evaluation first, the way test-driven development writes the test first. Let the score gate the release.

Stated that way it sounds like a solved discipline. In practice, teams adopt the ceremony — a dataset, a judge prompt, a number in CI — and then never establish whether the number tracks anything a user would recognize as quality. An eval suite that has not been validated against human judgment is not a safety net. It is a second system that can be wrong, running unsupervised, with the authority of a passing build.

What EDD actually claims to be

The term has an academic anchor, which is worth knowing if you are proposing this to a governance committee. Researchers at CSIRO's Data61, working with UNSW and the Australian National University, published the reference treatment in November 2024 and have revised it twice since, most recently in November 2025.

Their argument starts from a specific claim about why conventional methods fail. Test-driven and behaviour-driven development assume deterministic systems with predefined specifications. LLM agents pursue under-specified goals, behave probabilistically, and continue to change after deployment — sometimes without any code or model change at all. Fixed benchmarks and static test suites cannot capture emergent behaviour or support adaptation across a lifecycle.

The paper's answer is EDDOps: a process model and reference architecture that treat evaluation as a continuous governing function rather than a terminal checkpoint, unifying offline development-time evaluation and online runtime evaluation in a closed feedback loop. The load-bearing idea is that evaluation evidence should drive both runtime adaptation and governed redevelopment — architectural changes, memory updates, test-suite revisions — not just prompt tweaks and retraining.

Read the definition carefully, though. In that framework evaluation is an umbrella term covering testing, benchmarking, validation, online monitoring and risk assessment. It is a process model, not a metric. Anyone claiming EDD gives them a number that means "ready to ship" is quoting a framework that does not say that.

The practice: error analysis before evaluators

The industrial methodology has converged on an ordering that is the opposite of what most teams do, and it comes largely from Hamel Husain and Shreya Shankar, whose evals course has trained several thousand engineers and PMs including at the major labs.

The instruction is to start with manual error analysis, not with writing evaluators. Read real production traces. Write free-form notes on failures — open coding — which cannot be delegated to an LLM because it requires domain knowledge the model does not have. Group those notes into a taxonomy of a handful of failure modes — axial coding — where an LLM can help cluster once you have hand-coded enough traces. Count frequencies. Then, and only then, build targeted evaluators for the failures that actually occur.

The theoretical justification is a genuine finding rather than a slogan. Shankar and colleagues, in work published at UIST 2024, identified criteria drift: users need evaluation criteria to grade outputs, but grading outputs is how users discover their criteria. Evaluation is therefore an iterative human sensemaking process, not a static target you specify once and hand to an optimizer.

That single result undercuts the most common way EDD gets sold internally — "we define the evals up front, then engineering optimizes against them." You cannot fully define them up front. The people selling a clean TDD analogy are selling something the underlying research says is unavailable.

Two further practical positions worth flagging, because they are contrarian and they are held by the people with the most reps: prefer binary pass/fail judgments to 1–5 Likert scales, on the grounds that the added resolution creates more disagreement than insight; and treat generic off-the-shelf metrics with suspicion, since evaluators should emerge from observed failure patterns rather than a vendor's default list.

The judge problem, stated fairly

Most automated evals ultimately rest on LLM-as-a-judge. The foundational result here is genuinely encouraging, and it is often misquoted in both directions.

The MT-Bench and Chatbot Arena work found that strong judges matched both controlled and crowdsourced human preferences at over 80% agreement — the same level of agreement humans reach with each other. That is a real finding and it is why the technique is defensible at all.

The same paper documented the failure modes, and subsequent work has quantified them. Position bias: judges favour responses by their placement in the prompt. Verbosity bias: longer answers score higher regardless of quality. Self-enhancement bias: judges rate their own outputs more favourably. Figures circulating in the literature put position preference as high as 75% for the first-placed response and self-preference in the 10–25% range; treat those as indicative rather than universal, since they are aggregated across settings.

The most careful study on position bias is worth the read: a Dartmouth group ran 15 judges across MT-Bench and DevBench, 22 tasks and roughly 40 solution-generating models, producing over 150,000 evaluation instances. Their finding is more useful than a headline percentage. Position bias is not random noise, varies significantly by judge and by task, and is strongly affected by the quality gap between the candidates — meaning it bites hardest precisely where the comparison is close, which is exactly where your regression tests live.

The standard defences — reversing presentation order, ensembling judges, reporting inter-judge agreement — are real mitigations, but they address variance across the judge population rather than biases shared across it. And there is a structural gap that no vendor dashboard closes: LLM judges are used constantly to evaluate other systems while rarely being subjected to rigorous scrutiny themselves. If your eval suite has never been scored against human labels on a held-out sample, you do not know its accuracy. You have a number, not a measurement.

Benchmarks are not evals, and the leaderboards have documented problems

A common substitution is to point at public benchmark scores instead of building product-specific evals. Recent work makes that harder to defend.

The Leaderboard Illusion, a multi-institution study of Chatbot Arena, found undisclosed private testing that let a handful of providers test many variants and disclose only the best result — at the extreme, 27 private variants tested by one provider ahead of a major release. It also found proprietary models sampled at higher rates and removed less often than open-weight ones.

The mechanism finding is the one that should end the "we'll just use benchmark scores" argument. In controlled experiments, training on a high proportion of Arena data more than doubled win rate on an Arena-distributed benchmark — from 23.5% to 49.9% — while MMLU performance slightly declined. Leaderboard gains driven by selective data and testing do not necessarily reflect broader model quality. A public score tells you about a distribution that is not your users'.

The tooling market consolidated in 2026

This part is not a trend piece; the transactions are on record and they change procurement risk.

OpenAI announced on 9 March 2026 that it is acquiring Promptfoo, the open-source evaluation and red-teaming tool, with the technology to be integrated into OpenAI Frontier, its enterprise agent platform. Promptfoo reported more than 350,000 developers, 130,000 monthly actives and use at over 25% of the Fortune 500. Both parties committed to the project remaining open source under its current licence. Terms were undisclosed; the company had raised $23M and carried a roughly $86M post-money valuation from July 2025 according to PitchBook data cited in trade coverage.

Trade and independent review coverage also reports that Humanloop's standalone platform was sunset in late 2025 after Anthropic hired its founding team, and that Braintrust raised an $80M Series B at an $800M valuation in February 2026. Those are secondary sources, and worth confirming before either goes into a board paper.

The pattern matters more than any single deal. Evaluation tooling is being absorbed into the platforms whose models it evaluates. That is not automatically bad — the acquirers have both the traces and the incentive to invest — but a team that is committed to an eval platform now needs to ask a question that did not exist eighteen months ago: what happens to my evaluation evidence, and my ability to compare vendors with it, if the tool becomes part of one vendor's stack?

AI can build faster. Can your team decide better?

AI can draft the PRD and prototype the idea. Jira Product Discovery helps teams decide whether it belongs on the roadmap. Bring feedback and ideas together, prioritize as a team, and keep your roadmap connected to delivery in Jira.

Get Started for Free.

The fair objection: "we ship fine without this"

The strongest version of the objection is not laziness. It is that some very good teams ship frontier products with lightweight evaluation and heavy manual review, and say so publicly. The prominent "you don't need evals" position gets cited constantly in internal debates as permission to skip the work.

The rebuttal that holds up: those teams are doing intensive, continuous, expert manual error analysis — the expensive half of the discipline — and merely declining to formalize the cheap half into automated scorers. That is a defensible engineering trade-off when your reviewers are the domain experts and the product surface is small. It generalizes badly to a team of six shipping a customer-facing assistant with no one reading traces.

The real dividing line is not evals versus no evals. It is whether anyone is systematically looking at what the system actually produced.

The scorecard

The uncomfortable summary is that evaluation-driven development moves the hard problem rather than solving it. Previously you did not know whether your AI system was good. Now you have a number, and you do not know whether the number is good.

The step that separates the two is unglamorous and non-delegable: take a sample of real outputs, have a domain expert label them, and measure your evaluator against those labels. Report the agreement rate the way you would report test coverage. Teams that do this have an eval suite. Teams that do not have a dashboard, and dashboards fail silently — which is the most expensive way for anything to fail.

Don’t stop there

You cannot systematically evaluate an AI system if the data architecture beneath it is built on sand. As you move beyond fragile test suites and integrate models into production, your data platform must become the foundation for both grounding the agent and auditing its outputs.

When an LLM generates a high-scoring but fundamentally flawed response, can you trace exactly which definitions and source systems it used to get there? To move past silent dashboard failures, you need an architecture that supports deep traceability, historical context, and strict governance.

Before you finalize your evaluation metrics, ensure your underlying data foundation is actually built to support trustworthy, auditable AI:

• To master the traceability and context AI requires: Read Ignition Data’s guide on Understanding the Semantic Foundations That Make AI Work.

• To safely embed models into your enterprise pipelines without falling into the "garbage in, garbage out" trap: Discover Scalefree's practical breakdown in Unlock the Intelligence Layer: LLMs in Data Warehousing.

That’s a wrap for this week

Happy Engineering Data Pro’s