LLM Judges Do Not Have One Length Bias. They Have One Per Task

If you pick models by judged quality, the length of the winning answer is not a constant you can correct for. In our benchmark, judges picked the shorter answer on some tasks and the longer answer on others, and the direction flips even between closely related tasks. A single “penalize verbosity” or “reward thoroughness” adjustment will therefore get some of your tasks wrong.

The flip shows up between neighboring tasks

The clearest examples are pairs of tasks that sit next to each other in the same pipeline. Generating research queries and validating them are two halves of one job, yet the judges leaned in opposite directions on length. The same pattern repeats for subreddit work and for section writing.

None of these is a near miss. In every row the picked side differs from the other side by hundreds of tokens, and the direction reverses when you move one step along the same workflow.

The gaps can be large in either direction

The flip is not confined to tasks where answers are already similar in size. Extracting atomic fact claims favored the longer average at 6,299 against 2,926 output tokens [^10], more than double. Assigning sections to topic clusters went the other way, favoring the shorter average at 332 against 692 output tokens [^11][^12], less than half.

So the judge is not simply rewarding length when answers run long and brevity when they run short. The task decides what a better answer looks like, and length follows from that rather than driving it.

What this means when you choose a model

For production choices, length and judged quality move together differently by task. Work where shorter answers were picked, from query generation to translation to relevance scoring, will make concise models look better on judged quality. Work where longer answers were picked, from summarization to executive summaries to public response generation, will make verbose models look better.

The practical rule is to adjust for length task by task if you adjust at all, and never to carry a length assumption from one task to a neighboring one. The current evidence can support that kind of per-task adjustment; it cannot support one length correction applied across every task.

Even a clear winner usually wins by a hair

Length is one reason to read judged scores carefully. The size of the margins is another. On analyzing catalysts and scenarios, Claude Opus 5 was judged the best answer in 10 of the 30 occasions it competed in, 33.3%, every one of them against other models answering the same live input [^13]. But the top two answers scored exactly the same in 20.0% of 30 judged occasions, and finished within half a point in 73.3%, with a median gap of 0.2 on a 10-point scale [^14][^13].

On that task the picked answers were also longer on average, at 4,745 against 4,177 output tokens [^14][^13], so the length effect and the thin margins stack. Treat Opus 5 as a capable default there on this evidence, but expect ties and near-ties to be common, and decide routing on cost or operational fit unless a higher quality bar justifies chasing small gaps.

Where to look next

Check the trade-off for the quality bar you actually need on the live benchmark at https://llm-bench.kapualabs.com/, where you can compare batch against sync pricing for yourself. The per-task pages linked in the sources below, on llm-bench.kapualabs.com and fronset.ai, hold the figures quoted here. Method is documented at https://llm-bench.kapualabs.com/methodology/.