What Changes When the Bar Moves?

Raising your quality bar only buys a better model where the benchmark can tell the candidates apart. On visual theme configuration it can: Claude Sonnet 5 outscores GPT-5.6 Luna by more than the uncertainty in their scores, at about 26 times the cost per task run [^1][^2]. On metadata paragraph rewriting it can’t: the three models at the top are tied, so only cost separates them [^3].

How to read the scores and the bar

Each score is a single judge grading one output against a published rubric on a 0–10 scale, not a head-to-head comparison [^4][^5]. The ± figure after a model’s score is the 95% confidence interval on its mean quality, the margin of uncertainty around its average [^4][^5]. When two intervals overlap, the models are tied.

That uncertainty also decides whether a model is ranked at all. A model is ranked on a task only when its 95% confidence interval is ±0.2 points on the 10-point scale [^4][^5]. By that rule, 8,401 of 11,284 measured model-task results, 74%, are too uncertain to rank [^4][^5]. A missing rank means the measurements don’t settle the question, not that the model failed.

The bar itself is relative. A model qualifies on a task when its quality is within 90% of the best measured model’s, and that bar is raised to 95% on some tasks and lowered to 80% on others [^4][^5]. Because it is pegged to each task’s own best, a percentage bar does not translate into a fixed score that carries across tasks.

Where a higher bar separates the models

Visual theme configuration is the clearest case. Claude Sonnet 5 scores 9.41 out of 10 (±0.05) at $0.05 per task run, and GPT-5.6 Luna scores 9.01 out of 10 (±0.21) at $0.0019 per task run, both in sync (non-batch) mode and repriced at current rates [^1][^2]. The two intervals don’t overlap, so Claude Sonnet 5 is the stronger result on this task.

Below Claude Sonnet 5, the separation disappears. Thinking Machines Inkling Small scores 8.87 out of 10 (±0.27), which overlaps GPT-5.6 Luna, so those two are tied [^1][^2].

Where a tie leaves only cost

Metadata paragraph rewriting runs the other way. GPT-6 Sol scores 8.0 (±0.16) at $0.0038 per task run in sync mode, GPT-6 Luna scores 7.81 (±0.18) at $0.0002, and Xiaomi MiMo V2.6 Flash scores 7.81 (±0.22) at $0.0001 [^3].

GPT-6 Sol’s 8.0 is the highest of the three, but all three intervals overlap, so for a bar in this range they are equivalent on quality. Picking on the raw score alone would treat a tie as a lead.

What to do with this

Set the bar per task, against that task’s best measured score, and treat 90% as a starting point to adjust, not a universal cut-off [^4][^5].
Where a higher bar separates the leading models, pay for the stronger one only if the gap matters to your work.
Where intervals overlap, choose the cheapest model, comparing prices within one mode (batch or sync) and confirming it answers reliably on your own inputs.

The live benchmark at https://fronset.ai/benchmark/ lets you set the quality bar your task requires and see which models qualify.