High Quality Scores Often Come With Low Answer Rates
GPT-6 Sol scores 9.15 out of 10 on generating publication title packages, its highest quoted score, yet only 10% of its attempts at that work return an answer [^1]. On the Fronset benchmark, quality and answer rate are separate numbers, and the most impressive scores often sit next to the weakest answer rates.
What the quality score leaves out
The quality score is graded on the calls that came back. Attempts that produced nothing do not pull it down, so it tells you how good an answer is when you get one, not how often you will get one [^1][^2].
The pattern is not limited to one model. GLM-5.3 Flash scores 9.24 out of 10 on analytical writing that keeps its references intact, but answers on 52% of attempts, the smallest share for any work measured for it [^2]. Qwen 3.7 Plus scores 9.56 out of 10 on turning text into structured facts, on 40% of attempts [^3].
In a pipeline, the unanswered share is real work. For GPT-6 Sol’s title packages, the 9.15 describes one attempt in ten [^1]. The rest fall to a retry, a fallback model or a missing output, and none of that cost appears in the score.
One task, six models
Analytical writing that keeps its references intact shows the trade-off clearly, because results are quoted for six models on it [^2].
| Model | Score (out of 10) | Answer rate |
|---|---|---|
| GLM-5.3 Flash | 9.24 (±0.2) | 52% |
| GLM-5.3 | 8.88 | 50% |
| Xiaomi MiMo V2.6 Pro | 8.52 | 30% |
| Thinking Machines Inkling | 8.42 | 90% |
| Qwen 3.8 Max | 8.2 | 32% |
| Xiaomi MiMo V2.6 Flash | 8.18 | 66% |
Sorted by score, the answer rates go in no particular order. GLM-5.3 Flash is the measured quality leader, and it and GLM-5.3 each answer about half the time [^2]. Keep a fallback rather than making either the default.
Thinking Machines Inkling sits fourth on score but answers on 90% of attempts. It is the only result in its group where a high score and a high answer rate come from the same model on the same work [^2].
Model tier does not settle the choice either. Here Xiaomi MiMo V2.6 Pro scores higher and Xiaomi MiMo V2.6 Flash answers more often [^2]. On selecting which social posts to run, the order flips: Flash scores 8.66 out of 10 to Pro’s 8.41, but Flash answers on 9% of attempts and Pro on 73% [^4].
How to route on two numbers
Read together, the two numbers give three cases.
Both are high. Send the work there, especially when a miss is expensive: Thinking Machines Inkling for reference-preserving writing [^2].
One model scores higher, another answers more often. On short public-facing replies, GLM-5.3 scores 7.83 out of 10 at 62% of attempts and GLM-5.3 Flash 7.58 at 70% [^5]. If a failed attempt is cheap to retry, take GLM-5.3’s score; if not, Flash’s 70% decides [^5].
The peak arrives on a thin answer rate. Use it only with a fallback built. GPT-6 Sol’s title packages fit small batches where missing outputs are acceptable and a person reviews each answer [^1].
A higher quality bar pushes you toward the thin-coverage peaks. A requirement that the first call return pushes the other way, so decide which constraint binds before comparing scores.
Both numbers belong to the task, not the model
The caveat that matters is that neither figure transfers between tasks. GLM-5.3 Flash answers 52% of attempts on reference-preserving writing [^2] but 89% on choosing authoritative sources, where it scores 8.11 out of 10 [^6].
A model-level reliability figure is no substitute. Claude Sonnet 5.5 returns a usable answer first time on 98.1% of 4,214 attempts in its model-level results [^7], yet answers on 87% of attempts at short public-facing replies [^5]. Neither cancels the other. Use the per-task figure for the task you are routing.
A high answer rate is no proxy for quality either. NVIDIA Nemotron 3.5 Lightning answers 90% of attempts at picking authoritative sources while scoring 5.11 out of 10 [^6].
At https://fronset.ai/benchmark/ you can set the quality bar you actually need and compare the results for yourself; check each model’s answer rate on your task before trusting its score. The method is documented at https://fronset.ai/benchmark/methodology/.