A 9.78 Quality Score Means Less When the Model Only Answers 77% of the Time

If you choose a model from a leaderboard, check how often it actually returns an answer before you trust its score. On turning source text into structured facts, gemini-3.5-flash scores 9.78 out of 10, but it only answers on 77% of attempts, and that score is measured only on the calls that came back [^1][^2]. Qwen 3.7 Plus scores 9.67 out of 10 on the same work while answering on just 40% of attempts [^1][^2].

A quality score is conditional on getting an answer

The quality scores in this benchmark are computed from the responses a model produced. A declined or empty call is not scored, so it does not pull the average down. That is a fair way to measure how good an answer is when you get one, and a poor way to predict what your pipeline will ship, because every call that never came back still has to be handled by something.

The extraction task shows how far the two numbers can drift apart. The third strong model on this task, deepseek-v4-pro, scores 9.54 out of 10 when it answers but answers on 80% of attempts [^1][^2]. It ranks lowest of the three on answered quality and highest on completeness, so the model that looks weakest on the leaderboard is the one most likely to return something on a given call.

Narrow confidence intervals do not close the gap

It is tempting to read a tight interval as a sign that a score is safe to rely on. Qwen 3.7 Plus is one of the models ranked with most confidence on extraction, with an interval of ±0.15 after 31 independent judgments [^1][^2]. The interval describes how well the answered calls were measured; it says nothing about the six in ten calls that produced no answer.

The same pattern shows up at the top of other task lists. DeepSeek V4 Pro scores 9.86 out of 10 on identifying language when it answers, but it only answers on 57% of attempts [^3]. A near-perfect score on a task as simple as language identification would normally be an easy pick, and it still leaves close to half of the calls unresolved.

Answer rate changes with the task, not just the model

A single model can be reliable on one kind of work and mostly silent on another, so the answer rate has to be checked per task. DeepSeek V4 Flash scores 8.21 out of 10 on writing a short promotional post when it answers and answers on 88% of attempts [^4][^5]. On refining factual claims it scores 7.71 out of 10 but answers on only 36% of attempts [^6], and on extracting atomic factual claims it scores 7.43 out of 10 while answering on just 12% of attempts [^7].

That last figure is the important one. A model that answers roughly one call in eight is not a slightly worse option for claim extraction; it is a different product from the one that handles promotional copy, even though it is the same API endpoint. Cost per usable answer, not cost per call, is the number that matters when the answer rate falls that far.

Markdown repair offers a clean before-and-after on the same task. Qwen 3.7 Plus scores 5.87 out of 10 on repairing broken line breaks when it answers, and only answers on 28% of attempts [^8][^9], so it is weak on both dimensions at once. gemini-3.5-flash scores 9.33 out of 10 on the same work, ±0.14 after 39 independent judgments, yet still only answers on 65% of attempts [^8][^9].

What this means for routing

Treat any answered-only score as a ceiling, not a forecast. If you want gemini-3.5-flash or Qwen 3.7 Plus for high-precision extraction or markdown repair, put them behind retry or fallback logic, because non-answers rather than wrong answers will dominate your failures. Do not route markdown line-break repair to Qwen 3.7 Plus without an alternative path.

For DeepSeek V4 Flash, send promotional and similar generation work its way before factual-claim work, where its answer rate drops sharply. More generally, before committing a task to a model, run it through your own inputs and count how many calls return anything at all. That count, combined with the answered-only score, is what tells you how many calls you will really pay for per finished result.

Where to look next

The live benchmark at https://llm-bench.kapualabs.com/ lets you set the quality bar you actually need for a task and compare batch against sync pricing for your own workload. The method behind the scores, including how judgments and confidence intervals are produced, is documented at https://llm-bench.kapualabs.com/methodology/, and the per-task pages linked below show the answer rates alongside the scores.