The Top-Scoring Extraction Model Only Answers 40% of the Time

If you choose an extraction model by its quality score alone, you will overestimate what actually ships. On the structured-data benchmark every score is measured only on the calls that came back, and some of the best-scoring models come back far less often than you would expect. Qwen 3.7 Plus scores 9.67 out of 10 when it answers — but it only answers on 40% of attempts [^1][^2].

Two numbers, not one

Every quality score in this category is conditional: it describes the answers a model returned, not the requests you sent. A request that does not come back with an answer simply does not count against the score. That is fair for measuring quality, but it hides the second number that decides throughput, which is how often the model answers at all.

The structured-facts leaderboard makes the gap visible once both figures sit side by side.

Sorted by the middle column, gemini-3.5-flash wins. Sorted by the right column, deepseek-v4-pro is the most willing to answer among the high scorers, and Qwen 3.7 Plus pairs near-top quality with the lowest completion of the group. The ranking you get depends entirely on which column you sort by.

Why the Qwen figure is not noise

It would be tempting to treat the Qwen result as a small-sample fluke. It is not: the model is one of those ranked with most confidence on this work, at 9.67 out of 10, ±0.15 after 31 independent judgments [^1][^2]. The quality is real; it just applies to a minority of your requests.

The same model also shows that a conditional score does not transfer to neighbouring tasks. On repairing broken line breaks in markdown, Qwen 3.7 Plus scores 5.87 out of 10 when it answers — but it only answers on 28% of attempts [^3][^4], weak on both dimensions. A model near the top for structured facts can be uncompetitive on work that looks similar from the outside.

The trap gets worse on claim work

Structured extraction is the mild case. On extracting atomic factual claims, DeepSeek V4 Flash scores 7.43 out of 10 when it answers — but only answers on 12% of attempts [^5]. At that rate the score describes a small slice of your traffic, and a retry path stops being optional.

Averages can hide a second kind of weakness too. On refining claims, DeepSeek V4 Pro answered the same inputs as its rivals in 91 judged occasions and was chosen best in none of them [^6][^7]. No average quality score surfaces a result like that; only the head-to-head view does.

What to do with this

Treat the answer rate as a first-class routing input, not a footnote. Where a fallback or retry can absorb misses, the high-scoring but low-completion models remain the attractive choice for high-precision extraction, precisely because their misses are large and a second attempt is cheap insurance. Where a first attempt must return something, route to the higher-answer-rate model for that specific task and accept the slightly lower conditional score.

One caveat is worth stating plainly. This cycle’s evidence gives no prices, latency figures or confidence intervals for most cells, so it cannot name the cheapest model that clears a bar. What it can tell you is that the bar has two dimensions, and most model comparisons only show one.

Where to look next

The live benchmark at https://llm-bench.kapualabs.com/ shows both figures for every model and task, and the per-task pages linked in the sources below give the head-to-head results alongside the averages. Set the quality bar your workflow actually needs there, and compare batch against sync pricing yourself before you fix a production route.