Answer Length Sways Judges, and the Direction Depends on the Task
Judges in these benchmark measurements do not treat answer length neutrally, and which way they lean depends on the task. On scoring, detection, selection, repair and translation, the answer judges picked was generally shorter than the one they passed over. On analysis, summarizing, extraction and long-form writing, it was generally longer.
That matters for anyone choosing models by judged quality. The same writing habit can help a model on one task and count against it on the next.
How the split shows up
The comparison is simple: on the same inputs, take the average output length, in tokens, of the answers judges picked and of the answers they passed over. Each score is one judge grading one output against a published rubric on a 0–10 scale, and where one model is reported as beating another, both answered the same live input [^1].
| Task | Picked (tokens) | Passed over (tokens) | Favoured |
|---|---|---|---|
| Detecting a text’s language [^2] | 58 | 268 | Shorter |
| Translation [^3] | 1,150 | 1,699 | Shorter |
| Selecting posts for a social portfolio [^4] | 813 | 1,911 | Shorter |
| Analysing a document against a profile [^5] | 6,422 | 3,405 | Longer |
| Summarizing content [^6] | 3,333 | 2,203 | Longer |
| Analytical writing that preserves references [^7] | 10,628 | 6,924 | Longer |
The pattern is broad. Relevance scoring favoured the shorter answer in all four measurements reported [^8][^9][^10][^11]. Filing analysis and judging whether a community policy applies favoured the longer one [^12][^13].
The task category does not predict it
No single verbosity rule holds across the full set, and the exceptions follow the task, not noise. Extracting atomic factual claims from documents favoured the longer answer, 4,315 against 2,975 tokens [^14]. One claim refinement measurement went the other way, 1,561 against 1,762 [^15].
Promotional work splits too. Generating promotional campaigns [^16], promotional messages for a social platform [^17] and promotional copy for activities [^18] all favoured the shorter answer. A promotional post measured separately favoured the longer one, 1,695 against 1,225 [^19]. Even analysis is not uniform: catalyst and scenario analysis picked the shorter answer, 3,633 against 4,047 [^20].
The size of the gap varies as well. Naming topic clusters sits close, at 1,004 against 889 [^21], while one form of content-domain suggestion is far apart, at 2,257 against 574 [^22]. This part is interpretation, not a measured result: where the counts are close, length is probably a weak influence, and where they diverge, a model that writes at a different length is likely being sorted by length as well as by quality.
Reading a score with length in mind
Read a quality score together with the output-length pattern for the specific task you care about, not its broad category. On scoring, identification, translation and selection work, a verbose model is working against the judging, as on relevance scoring [^8] and portfolio selection [^4]. On document and filing analysis, summarization and extraction, a terse model will look weaker than its substance may warrant [^5][^6][^12].
In practice, check which way length cuts on your task before trusting a close result between two models whose answers differ noticeably in length.
The caveat that matters: the evidence cannot say how much of any score gap length accounts for, and it carries no model-level length figures. It shows where a score may be tilted, not by how much. A measurement that separates quality from length on the same task, or a quality bar that penalises verbosity explicitly, would change this reading.
The live benchmark at https://fronset.ai/benchmark/ lets you set the quality bar you need and compare models on the specific task you route, which is where the length pattern applies. The method is documented at https://fronset.ai/benchmark/methodology/.