Two numbers the leaderboard score is hiding

A benchmark score is one number measured under conditions your production system does not have. Two of those conditions matter more than the score itself. The score was measured only on the calls that came back, so it says nothing about the calls that did not. And it was measured without asking whether the second-best model would have done just as well on the same input, which is the question you are actually deciding.

We measure both on llm-bench, across the small tasks that make up a real content pipeline: identify a language, pull structured facts out of a document, label a topic cluster, summarize, write a teaser. Scores are out of 10 and come with a 95% interval and the number of independent judgments behind them. This is what the two hidden numbers look like.

The score only counts the calls that came back

Gemini 3.5 Flash turns source text into structured facts at 9.78 out of 10. That is one of the best extraction results we have. It also answers on only 77% of attempts, and the 9.78 is measured on the calls that came back [^1][^2]. On repairing broken line breaks in Markdown the same model scores 9.33 (±0.14, n 39) on a 65% answer rate [^3][^4]. Qwen 3.7 Plus shows the same shape more sharply: 9.67 (±0.15, n 31) on structured facts with a 40% answer rate [^1][^2], and 5.87 (±0.2, n 46) on line-break repair with a 28% answer rate [^3][^4].

Neither number is wrong. A 9.67 is what you get when Qwen 3.7 Plus answers, and if you have a retry loop and a fallback model in front of it, that may be what you experience. Without them, 40% is what you experience. If the misses were independent, a 40% answer rate means two and a half calls per completed task before you have written any of the retry logic.

The pattern is not confined to one vendor or one size of model.

Scores are out of 10, measured only on calls that returned an answer. Answer rate is the share of attempts that returned one at all.

What makes this actionable is that on most of these tasks there is a model with no gap at all. On structured extraction, GPT-5.6 Sol scores 9.74 (±0.13, n 34) with no answer-rate caveat [^1][^2], within a rounding error of Gemini 3.5 Flash’s 9.78. On language identification, DeepSeek V4 Pro’s 9.86 on a 57% answer rate sits below Claude Sonnet 5 at 10.04 (±0.02, n 125), GPT-5.6 Sol at 10.0 (±0.02, n 50), Claude Haiku 4.5 at 9.92 (±0.04, n 324) and GPT-5.4 Nano at 9.92 (±0.04, n 392), none of which carry one [^16][^13]. On topic cluster labeling the choice is a real trade-off: Claude Opus 5 scores 8.91 when it answers but answers on 76% of attempts [^10], while Claude Sonnet 5’s 8.38 (±0.19, n 45) comes with no answering gap [^9][^10].

Answer rate is also not the only way a call fails to arrive. Against the 600 seconds we allow a synchronous call, GPT-5.6 Sol misses on 25.6% of its 4,061 attempts [^17][^18][^19] and GPT-5.4 Mini on 14.8% of its 2,413 [^20][^21]. Looking at why calls miss, no single cause dominates: timeouts 18.0%, rate limiting 19.1%, parse failures 12.0%, our own misconfiguration 19.2% [^20][^21]. Two of those four are on the caller, not the model. At the other end, NVIDIA Nemotron-3 Nano 30B-A3B returns a usable answer, first time, on 99.3% of its 106,736 attempts [^22][^23][^24][^25]. Its scores are middling, 7.82 (±0.21, n 215) on factual claim refinement [^26][^8], and for a pipeline that cannot afford retries a 7.8 that always arrives can be worth more than a 9.5 that arrives four times in ten.

The runner-up was usually just as good

The second hidden number comes from how the scores are produced. Each judged occasion is one real input answered by several models, with every answer scored. That lets us look at the gap between the two best answers on the same input. On most tasks there barely is one.

Median gap is between the two best answers on the same input, on the 10-point scale.

Read that as a routing table. When the top two answers finish within half a point nine times out of ten, the quality axis is exhausted and the decision moves to price, latency and the answer rate above. The spread on those is not thin. GPT-5.4 Nano generates publication title packages at 8.04 (±0.23) for $0.0015 per task run in batch mode, measured on the model’s own usage and repriced at current rates [^44][^40]. Gemini 3.5 Flash Lite detects a language in 1.0 seconds at the median over 57 synchronous calls [^27][^13][^45][^28], while Qwen 3.7 Plus takes 215 seconds at the median to repair Markdown line breaks over 46 [^20][^27][^4]. GPT-5.4 Nano summarizes content in 12 seconds at the median but 38 seconds at the 90th percentile over 1,390 calls [^42][^27][^43], which is the kind of tail a tied quality score will never warn you about.

The thinness also cuts the other way. A model can post an ordinary average and never once be the best answer. GPT-5.4 Nano was judged best in 28 of 71 content-summarization occasions, 39.4% [^42][^43], and in 0 of 50 engagement-triage occasions [^34][^35]. No average surfaces a shutout; only the head-to-head does.

Where the gap is real

The exceptions are the open-ended judgment tasks, and they are where model choice still earns its evaluation budget.

On these tasks the head-to-head swings are large and model-specific. GPT-5.6 Sol was judged the best answer in 25 of 29 subreddit selections, 86.2% [^52][^53], and in 10 of 30 content summarizations, 33.3% [^42][^43]. Thinking Machines Inkling Small won 28 of 33 content-domain suggestions, 84.8% [^60], on the same work where Tencent Hy3 answered 41 judged occasions and was never chosen best [^60]. Meta Muse Spark 1.1 won 39 of 45 domain suggestions, 86.7% [^59][^60], and 13 of 42 executive summaries, 31.0% [^61][^62]. Content domain suggestion is also where absolute scores collapse: GPT-5.6 Sol scores 3.84 (±0.1, n 30) [^63][^58], and Grok 4.5’s 4.66 (±0.24, n 30) is the best of the six models scored on it [^63][^58]. A wide gap between the top two is not a sign that someone is good at the task. Sometimes it means everyone is bad at it in different ways.

What to do with the two numbers

Read the answer rate next to the score. If it is much below 90%, either the retry-and-fallback design is part of the cost of that model or you pick the model without the gap, and on extraction, identification and matching there is almost always one.

For tasks where the top two tie, which in our data is extraction, identification, matching, refinement, translation, titling and summarization, stop comparing quality and compare price, median and tail latency, and first-time answer rate. Those spreads are wide where the quality spread is not.

Spend evaluation effort where the gap is real: domain suggestion, query validation, metadata paragraph improvement, subreddit selection, theme and query generation. There the leader changes by task, wins are lopsided, and a good average can hide a shutout.

Check the live benchmark at https://llm-bench.kapualabs.com/ for answer rates, top-two gaps and batch against sync pricing on the task closest to yours before you route production traffic.