First-Shown Answers Score Higher, so Small Gaps Are Noise
First-Shown Answers Score Higher, so Small Gaps Are Noise
On the Fronset benchmark, the order in which a judge sees candidate answers moves the score. Across 105,928 position-stamped judgments, the candidate shown first averages 7.82 and the one shown last averages 7.55 [^1]. A gap between two models that is smaller than that 0.27-point drift is weak evidence that either one is better.
Why answer order leaks into a score
An LLM judge comparing several candidates reads them one after another, and its verdict is not fully independent of that sequence. Unless stronger answers were somehow shown first more often, the higher average for the first slot comes from the judging, not from the models.
This error behaves differently from ordinary scatter. Random noise shrinks as judgments pile up, and a confidence interval tracks how far it has shrunk. A position effect is systematic: more judgments measure it more precisely but do not make it smaller. It drops out of a comparison only when each model is shown first and last about equally often.
The benchmark does measure the biases it knows about. Besides the position stamps, it holds judge bias as 1,389 measured offsets across a 64-judge pool [^1]. What the material does not state is how either the position drift or those offsets is applied to a quoted score [^1]. A reader cannot tell from a published number whether it has been corrected.
A tight interval does not rule it out
A model is ranked only once the 95% interval on its mean is ±0.2 points on the 10-point scale [^1]. The first-to-last drift is larger than that margin.
The two measures answer different questions. The interval says how much a score might move with more judgments of the same kind. The drift says how much the same answer can move by being shown in a different slot. A cell could meet the precision bar and still carry an order effect if its model sat in one slot more often than its rivals did. The material does not say that happens, only that the correction is not described.
Which published gaps survive
Three pairs from the current results show how the drift separates real differences from noise.
| Models | Work | Scores | Gap against the drift |
|---|---|---|---|
| Thinking Machines Inkling vs Inkling Small | Synthesizing a decision from several perspectives | 8.56 ±0.25 vs 8.45 ±0.22 [^2] | Smaller |
| Tencent Hy4 Preview vs Thinking Machines Inkling | Judging claim consistency | 7.57 vs 7.38 [^3] | Smaller |
| Gemini 3.8 Flash vs Gemini 3.5 Flash Lite | Building an author’s voice profile | 9.1 ±0.17 vs 7.73 ±0.28 [^4] | Several times larger |
The Inkling pair fails twice over: the intervals overlap and the gap sits inside the drift. The claim-consistency pair has no intervals quoted, so the drift is the only yardstick, and it says the gap is too small to read. The Gemini pair is the opposite case. Its intervals are well apart, and the gap is several times what moving an answer from the first slot to the last is worth on average.
What to do with it
- Check a gap against the drift before acting on it. Below it, treat the two models as indistinguishable on quality and test both on your own inputs.
- Assume a published score may be uncorrected for order until the method says how corrections are applied.
- Control order in your own LLM-as-judge evaluations. Record which slot each answer was shown in, rotate the order so every candidate appears first as often as last, and compare average scores by slot to measure your own drift.
The caveat that matters is that the position drift is an average over the whole pool. The material gives no breakdown by task or judge, so a particular cell could carry more or less order effect than the average. Answer length is a second judging effect, separate from position.
To read a gap with this in mind, open the task you care about at https://fronset.ai/benchmark/, find the models you are choosing between, and compare the distance between their scores with the position drift before reading anything into it.