Who Watches the Judges? (6 of 11)

Here’s the objection that should occur to anyone reading a benchmark like ours, and that we’d rather raise ourselves than have you raise for us: you use LLMs to grade LLMs. So who judges the judges?

It’s a fair hit. A score is only as trustworthy as the thing that produced it, and if the grader is the same kind of system as the contestant — prone to the same blind spots, the same stylistic preferences, the same overconfidence — then the whole leaderboard is just one model’s taste dressed up as measurement. A benchmark that waves this away with “our judge is very capable” hasn’t answered the question. It’s hoped you won’t ask it.

This is the evaluation layer of the harness — the ninth component of the machine we’ve been describing across this series, the complete task-specific infrastructure that surrounds a model to make it reliably do a defined job. Evaluation is the part that decides what “reliably” even means: it’s how the harness knows which model to trust for the work, and how it keeps knowing as models change underneath it. And that layer is only as good as its judges. So this post answers the objection, mechanism by mechanism. The short version: the reliability of the evaluator is itself something you engineer and measure — the same philosophy the harness applies to the models on the heatmap. The longer version is five guardrails and one habit. The habit is publication: we put the judge panel on the bench, dated, so you don’t have to take any of this on faith. (For where these scores come from in the first place, see Measuring quality; this is the companion piece — not how the harness grades, but why you can trust who grades.)

See it live: the judge panel is published in full on the bench — open it and check who’s doing the grading before you trust the grades.

No single judge decides anything

The first and biggest fix is structural: no single judge produces a score. A score pools many verdicts, from several strong models from different labs, each grading against the same published rubric without seeing the others’ scores. How many judges read one output depends on the route. When a sampled production call is graded, the whole panel grades it and the harness averages their scores. When models are compared on the same live input — which is where most of the grading happens — one judge from the panel grades all of the candidates, and different comparisons draw different judges.

Then the harness averages: the mean of the verdicts, each corrected for its judge’s strictness (below). The point is variance reduction. Every model has quirks: one over-rewards fluent prose, one is a stickler about format, one is a soft touch on factual slips. Pool verdicts from a diverse panel and the idiosyncratic noise starts to cancel, while the real signal — is this output actually good? — survives. No single judge’s bad day, or pet preference, can swing a published score on its own — though a single judge does decide each comparison it grades.

This is also why a wider panel doesn’t just produce a different number — it produces a steadier one. More independent reads, less sampling jitter. That steadiness matters downstream, and we’ll come back to it.

Where a model’s own judge is left out — and where it isn’t

The most glaring conflict of interest in LLM-as-judge is the obvious one: a lab’s model grading that lab’s own output. Even with no ill intent, models from the same family share training lineage, share stylistic instincts, share failure modes. A judge that “recognizes” its sibling’s style as good is not a fair referee — it’s a character witness.

So when the whole panel grades a production call, the judge whose own model produced the output is left out of that grading. The rule is per model, not per lab: a judge from the same lab can still grade a sibling model. And when one judge grades a comparison, the rule does not apply at all — that judge can be grading its own model’s output, and the verdict counts like any other. Self-preference is not corrected for; the methodology says so in as many words.

We state this plainly because it is the weakest point of the arrangement. What limits the effect today is dilution: a published score pools verdicts from several judges from different labs, so any one judge’s preference is averaged with the others’. Dilution is not removal, and the published panel lets you see which labs have a judge — and so which models can be graded by a judge of their own family.

Lenient and harsh judges, corrected before they count

Even honest, independent judges aren’t neutral instruments. Run enough gradings and the personalities emerge: one judge is consistently lenient, handing out eights where the panel says six; another is consistently harsh, docking points everyone else waves through. If you just average raw scores, the lenient judge quietly inflates every verdict it touches and the harsh one drags things down — and which models happen to draw which judges starts to matter more than it should.

So the harness measures each judge’s systematic tendency and corrects for it before aggregating. Think of it as a per-judge zero-point adjustment: if a judge runs half a point hot on a given kind of task, that bias is subtracted back out before its score joins the panel average.

Two details matter. First, the correction is per judge, per task — not one global knob. A model can be lenient on long-form writing and dead-on for structured extraction; a single blanket adjustment would break one to fix the other. Second, it’s an offset, not a thumb on the scale: what’s removed is a judge’s measured tilt — how far its average sits from the panel’s average on the task — so its ability to tell good from bad is untouched. One limit: the tilt is measured over whatever outputs the judge happened to grade, so a judge that drew weaker outputs can look stricter than it is. And none of this is calibration against human labels; there are none.

Judges are watched for drift

The strictness correction handles the biases you can see today. But a judge panel is a living thing — model endpoints get updated, providers change defaults, and a grader that behaved one way last month can quietly start behaving differently. An uncorrected drift is more dangerous than a static bias, because a static bias gets corrected and a drift just accumulates.

So the judges are supervised, every night, the way the harness supervises any production component it depends on. It tracks how often each judge agrees with its peers, how often it returns no usable score, and whether its average is drifting against its own recent baseline. A judge that crosses the alert thresholds is flagged to an operator; one that crosses the severe thresholds is taken off the panel automatically, unless that would leave its lab with no judge — then an operator is alerted to replace it by hand.

The principle is the one the harness applies to everything on the bench: a number you can’t see going wrong is a number you can’t trust. Watching the watchers isn’t a one-time audit — it’s a standing measurement, with the judges held to the same “prove it, keep proving it” bar as the models they grade.

Changing a judge is the riskiest edit we make — so we rehearse it

Here’s a subtle danger that’s easy to miss. Swapping a judge model — retiring one, promoting a new one onto the panel — is one of the most destabilizing edits the harness can make. A new judge with even a slightly different temperament could shift every score on the board at once, silently, and there’d be no clean way to tell a real ranking change from an artifact of the new referee.

So we don’t just swap a judge and hope. Before replacing one, we can replay past gradings through the candidate on a shadow panel — re-running historical work it never saw, and comparing its verdicts against what the current judge concluded: how closely they agree, how far apart they sit, and how many published bands the swap would move. That replay is a tool we run before a swap, not a gate the system enforces: what the system requires before a new panel goes live is that the panel’s own measurements have reached the required confidence.

And — the habit again — when the panel does change, the snapshot records the new composition and the date it took effect, and the methodology page shows the current panel. A published score pools the verdicts given over time, so a score can include verdicts from judges that have since been retired.

We publish the panel — that’s the whole point

Every guardrail above is, in the end, in service of one thing you can do yourself: look.

The bench publishes a dated snapshot of the judge panel — which models are grading, in what roles, as of that snapshot. Not “a panel of frontier models.” The actual composition, frozen with the rest of the snapshot, next to the rubrics and the example inputs and outputs. If you think a particular grader shouldn’t be on the panel, you can see that it’s there and argue the point. If you want to know which labs have a judge on the panel — and so which models can be graded by a judge of their own family — you can check. The answer to “who watches the judges?” is, ultimately, you can — because we showed you who they are.

That’s the difference between “trust us” and “audit us,” and it’s the only honest stance to take here. A lab harness — a fixed set of curated prompts, run once, scored, and ranked — never has to answer this question, because nobody depends on its judges past the moment the ranking is published. A production harness runs messy, single-pass work all day, and its evaluation layer is load-bearing: get the judges wrong and every routing decision downstream inherits the error. So the bench is the readout of the harness’s judges — a dated snapshot of the evaluation layer, held to the same standard as the work it governs. Let’s be candid: LLM-as-judge is an active, imperfect, fast-moving area of practice. We are not claiming the judges are flawless, and we have not calibrated them against human labels. We’re claiming something narrower and more defensible — that the harness treats the evaluator as a system to be engineered and measured rather than assumed: a panel from several labs, a model’s own judge left out where the whole panel grades, strictness corrected, drift watched, judge swaps replayed first, and the whole arrangement published so you can check our work — including where those rules stop. A benchmark is only as trustworthy as its judges. So we built the judges to be inspected.

There’s a thread running out of this post worth pulling. Notice that nearly every guardrail here — pooling verdicts across the panel, the strictness offsets — does the same quiet thing: it reduces variance. Less noise in the grading means a tighter interval around each model’s score, which is exactly what lets a verdict graduate from “we think” to “we’re sure.” That machinery of confidence — why every score on the bench travels with an interval, and why the harness hides the cells it’s not yet sure of — is the next piece. And it’s the same panel, run at volume, that places a brand-new contestant on the scale for the first time in Onboarding a new model.

Next in the series: Confidence, not just scores — why a number without an interval is a guess wearing a lab coat. Open the heatmap and check who’s grading before you trust the grades.