Answer Coverage and Conditional Quality
In the benchmark, GPT-6 Sol scores 8.98 out of 10 on publication title package generation, but answers on only 10% of attempts [^1]. A quality score tells you how good a model’s answers are, not how often you will get one. Choose on the score alone and you can end up with a model that returns nothing for most of your requests.
What the quality score leaves out
The benchmark scores quality, out of 10, only on the calls a model answered. The answer rate, the share of attempts that came back with an answer, is a separate figure. Across the results here, it runs from 9% to 90% [^2][^3], and for many models the answered calls are a minority of attempts.
On two tasks, the high scorers rarely answer
GPT-6 Sol is not an isolated case. On the same task, GPT-6 Luna scores 8.61 out of 10 and also answers on 10% of attempts [^1]. Nemotron 3.5 Lightning scores 7.45 out of 10 with answers on 85% of attempts [^1], the combination a production pipeline can use without relying on a fallback for most requests.
Catalyst and scenario analysis shows the same split. GPT-6 Sol scores 8.78 out of 10 and GPT-6 Luna scores 8.6, and both answer on only 9% of attempts [^3]. MiMo V2.6 Pro scores 7.72 out of 10 with an 82% answer rate [^3], a far more usable profile.
When the higher score and the higher answer rate belong to different models, as on both these tasks, the choice turns on whether a better answer or a more frequent one matters more. The scores alone can’t make that call.
Almost the same score, very different answer rates
On claim continuity classification, Moonshot Kimi K3 scores 8.66 out of 10 and answers on 70% of attempts [^4]. Claude Opus 5.5 scores 8.64 but answers on only 19% of attempts [^4]. On score alone they look like near equals, yet only one of them answers on most attempts. Claude Opus 5.5 should not be the only path to usable output on this task.
Unanswered requests need a plan
On reference-preserving analytical writing, GLM-5.3 Flash scores 9.22 out of 10 but answers on only 52% of attempts [^2]. It is worth considering only where a team can accept missing responses on roughly half of requests.
Even a model that answers most of the time leaves a gap. On structured output extraction, gemini-3.5-flash scores 9.78 out of 10 and answers on 77% of attempts [^5][^6], and it still needs a retry or fallback path for the attempts that return nothing.
What to do with this
- Put the answer rate next to the quality score in every model comparison you run or read.
- Compare models only within the same task; quality figures from different tasks are not a league table.
- Set the lowest answer rate each task can tolerate before you compare scores.
Figures will move as the measurement continues; the live benchmark at https://fronset.ai/benchmark/ lets you set the quality threshold you need, and the method is documented at https://fronset.ai/benchmark/methodology/.