Read the Interval, Not the Judgment Count
Read the Interval, Not the Judgment Count
A score backed by more judgments is not necessarily a firmer score. On the Fronset benchmark, 129 judgments leave MiniMax M3’s result on synthesizing several perspectives into a decision at ±0.14 [^1]. Qwen 3.7 Plus gets to ±0.13 on fixing broken line breaks in Markdown with 46 [^2].
Why the count is a poor proxy
An interval narrows as judgments accumulate, but its width also depends on how much those judgments disagree. Where judgments cluster tightly, a score settles quickly. Where they scatter, many more are needed to reach the same width.
The evidence behind this article does not report that spread for each task. So the count alone cannot tell you which case you are in.
The four results that report both a judgment count and an interval show how loosely the two track:
| Model | Work | Judgments | Interval |
|---|---|---|---|
| GPT-6 Sol | Title packages | 30 | ±0.23 [^3] |
| Qwen 3.7 Plus | Fixing Markdown line breaks | 46 | ±0.13 [^2] |
| GLM-5.3 Flash | Structured content summarization | 53 | ±0.18 [^4] |
| MiniMax M3 | Synthesizing perspectives into a decision | 129 | ±0.14 [^1] |
GLM-5.3 Flash has more judgments than Qwen 3.7 Plus and a wider interval. MiniMax M3 has the most judgments here, yet its interval is no tighter than Qwen 3.7 Plus’s. Only GPT-6 Sol fits the intuition that few judgments mean a loose figure.
What the interval tells you
The benchmark ranks a model on a task only once the 95% interval on its mean quality has narrowed to ±0.2 points on a 10-point scale [^5]. GPT-6 Sol scores 9.15 on title packages, with an interval of ±0.23 [^3]. That is wider than the ranking bar [^5], so an eye-catching score is also the least settled figure in the table.
Firmness is not the same as quality. On suggesting content domains, four models score between 3.49 and 3.94, with intervals between ±0.12 and ±0.19 [^6]. Those scores are measured firmly, and they are firmly weak.
Reading the interval also settles close calls. On writing voice profiles, Claude Sonnet 5.5 scores 8.98, ±0.24 and Claude Haiku 5.5 scores 8.8, ±0.21. The intervals overlap, so the higher figure does not make Sonnet 5.5 the better model [^7].
Most of the map is still blank
Of 16,582 measured model-task cells, 11,182 (67%) are still too uncertain to rank [^5]. An unranked cell is not evidence of anything about the model [^5]. It is neither a hidden weakness nor a hidden strength. It is a result that has not settled.
The evidence also does not say how firmly the remaining third is pinned, cell by cell. So do not assume that every published number outside the unranked two-thirds is equally solid.
How to read a score
- Start with the interval. Compare it with the ±0.2 ranking bar before you look at the judgment count.
- Treat a wide interval as provisional. That holds however many judgments sit behind it. Treat overlapping intervals as a tie.
- Read “unranked” as missing evidence. If an unranked model matters to you, test it on your own material rather than ruling it out.
One caveat matters more than the rest. An interval describes only the calls that returned an answer. Qwen 3.7 Plus’s ±0.13 is the tightest interval in the table, yet it answers on 28% of attempts [^2].
At https://fronset.ai/benchmark/ you can open any task and check the interval behind each score before you treat a number as settled enough to build on.