Fronset Benchmark

Scores You Can — and Cannot — Trust Yet

A benchmark score needs two numbers beside it: its confidence interval (how precise it is) and its answer rate (how often the model answered). In the benchmark, 74% of 11,284 measured model-task pairs are too uncertain to rank [^1][^2], and some impressive scores rest on a small share of attempts. Read alone, a score can make a tie look like a lead or hide a model that mostly doesn’t answer.

What it takes to rank a model on a task

Each score is one judge grading one output against a published rubric on a 0–10 scale [^1][^2]. The benchmark ranks a model on a task only when the 95% confidence interval on its mean score is within ±0.2 points [^1][^2].

A missing rank means the evidence hasn’t settled the question, not that the model is poor [^1][^2]. Benchmark-wide volume doesn’t change that: the ranked scores rest on 239,838 judgments of real production work, but the total doesn’t make any single result safer than its own interval [^1][^2].

Overlapping intervals turn apparent leads into ties

A high score with a wide interval is easy to over-read. On Batch Text Translation, GLM-5.3’s 8.83 out of 10 (±0.34) is above GPT-5.6 Luna’s 8.74 (±0.18), but the intervals overlap, so the quality difference is not established [^3][^4].

Tight intervals don’t guarantee a clear winner either. On language identification, Nemotron-3 Ultra 550B scores 9.99 (±0.05), Nemotron-3 Super 120B 9.94 (±0.11) and Nemotron-3 Nano 30B-A3B 9.91 (±0.08) [^5][^6]. The intervals overlap, so the benchmark cannot show that Ultra is better than Super or Nano on this task.

High scores, low answer rates

A quality score leaves out attempts that didn’t return. GPT-6 Sol’s 8.98 out of 10 on Publication Title Package Generation is measured on only 10% of attempts [^7], and its 8.78 on Catalyst And Scenario Analysis on only 9% [^8]. Its most usable high score is a lower one: 8.54 on Visual Theme Configuration Generation, with an 84% answer rate [^9].

The pattern isn’t limited to tiny answer rates. On structured output extraction, Moonshot Kimi K3 scores 9.6 when it answers but answers on 76% of attempts [^10], and MiniMax M3 scores 9.36 but answers on 58% [^10][^11]. Both are strong scores on a partial population.

A quality gap doesn’t fix a low answer rate either. On claim continuity classification, Claude Opus 5.5 scores 8.64 and Thinking Machines Inkling scores 7.37 on the calls they answer, but they answer on only 19% and 20% of attempts respectively [^12]. Neither model can be the only source of usable output on that task.

What to do with this

  • Don’t rule out a model just because it has no rank on your task.
  • Treat models whose intervals overlap as tied, whatever the point scores say.
  • Unless a model’s answer rate makes missing answers rare, build and test a retry, fallback or review path before relying on its score.
  • Until more evidence arrives, use a model with a wide interval, a tied score or a low answer rate on your task for testing or as a fallback, not as your primary choice.

Check the live benchmark at https://fronset.ai/benchmark/ for the current interval and answer rate on the exact task you plan to run.