Relevance Scores Depend on How the Job Is Framed
Relevance Scores Depend on How the Job Is Framed
A model that judges the relevance of a single piece of content well can score far lower when asked to score a whole set, and the model at the top changes with the framing. On this benchmark, relevance scoring is three different jobs with three different ceilings.
That makes the framing your product uses a design choice with measurable consequences. A relevance leaderboard measured on one framing tells you little about another, so benchmark the one you will ship.
Three framings, three ceilings
The benchmark measures relevance three ways: judging one piece of generated content, scoring a whole set of content, and scoring batches of short posts. Scores are out of 10.
| Framing | Top score | Where the rest sit |
|---|---|---|
| One generated piece | Qwen 3.7 Plus, 8.73 (±0.34) [^1] | GPT-6 Luna at 8.13 (±0.38) still overlaps the top [^1] |
| Whole content set | DeepSeek V4.1 Flash, 6.81 (±0.28) [^2] | Down to 1.74 (±0.46) for NVIDIA Nemotron 3.5 Lightning [^2] |
| Short-post batches | MiniMax M3, 7.43 (±0.12) [^3] | Most measured models in the mid-6s [^3] |
Single pieces behave almost like fixed-label work: scores are high and the leaders’ margins overlap. Whole sets have the lowest ceiling of the three, and short-post batches sit between the two. On both multi-item framings, no measured model is strong.
The framings also differ in how often the two best answers score the same. On single generated pieces, the top two tied exactly in 63.4% of 41 judged occasions [^1]. On short-post batches they tied in only 13.6% of 59, with a median gap of 0.3 points [^3].
The ranking moves with the framing
Models measured under more than one framing do not keep their place:
- DeepSeek V4.1 Flash trails Qwen 3.7 Plus and Grok 4.7 (8.53, ±0.49) on single pieces at 8.22 (±0.41), within overlapping margins [^1], yet has the top whole-set score [^2].
- MiniMax M3 has the top short-post score [^3] but scores 5.58 (±0.28) on whole sets [^2].
- Claude Opus 5.5 scores 8.24 (±0.39) on single pieces [^1], 6.18 (±0.4) on whole sets [^2] and 6.45 (±0.4) on short-post batches [^3].
- Claude Haiku 5.5 scores 5.95 (±0.39) on whole sets [^2] and 6.61 (±0.33) on short-post batches [^3].
Not every model moves. Meta Muse Spark 1.3 stays near the top of both multi-item framings, at 6.7 (±0.38) on whole sets [^2] and 7.0 (±0.4) on short-post batches [^3].
The caveat that matters: these are different jobs on different inputs. The gaps show that a model’s strength depends on what it is asked to judge. They are not a ranking of the same models under identical conditions, and the benchmark does not measure why a model gains or loses between framings.
What to do with it
Benchmark the framing you will ship. If your pipeline scores many items together, choose from the whole-set or short-post results, not the single-piece leaderboard. When the framing changes later, re-check the model too.
If you control the framing, run the alternatives on your own data. The benchmark cannot tell you whether splitting a set into single judgments keeps single-piece quality, because its tasks use different inputs.
Put a review step in front of multi-item output. With no strong model on whole sets or short-post batches, treat those scores as a draft and test candidates on your own inputs if your bar is higher.
The live benchmark at https://fronset.ai/benchmark/ lists each relevance framing as its own task. Open the one that matches how your pipeline asks the question, set the quality bar you need, and compare the models measured on that framing.