What We Measure, and What We Don't
Each figure the benchmark publishes answers one narrow question. A quality score covers one model on one task, not the model in general, and a latency median covers the typical call, not the slow ones. Read either as a broader verdict and it can mislead you on which model to pick or what latency budget to set.
A quality score belongs to one task
Grok 4.5 scores 8.94 out of 10 (±0.09) on topic sequence optimization [^1][^2] and 4.66 out of 10 (±0.24) on content domain suggestion [^3][^4]. The ± figure is each score’s stated interval, its margin of uncertainty. A strong score on one task does not carry over to another.
A score on a similar-sounding task is no safe substitute either. Topic sequence optimization and topic sequence determination look related, but they are not shown to be the same work.
And not every model is measured on every task. When these figures were published, the benchmark covered 27 models on 54 tasks through 906 measured model-and-task combinations, so the score you need may not exist.
Overlapping intervals mean a tie
Even on the same task, scores support a comparison only where their intervals don’t overlap. On content-to-image prompts, GPT-5.6 Sol scores 9.13 out of 10 (±0.14) and Thinking Machines Inkling scores 9.07 out of 10 (±0.26) [^5][^6]. The intervals overlap, so the benchmark cannot tell the two apart there.
A latency median hides the slow calls
The latency figures here are medians over synchronous calls, where the caller waits for each answer. On engagement triage, Nemotron-3 Super 120B has a median of 21 seconds [^7][^8][^9][^10][^11], but its 90th percentile, the time within which nine in ten calls finished, is 94 seconds across 278 calls [^7][^8][^10].
A latency budget set on that median would leave a share of calls running well past it. Few of the results here report a percentile at all, so for most tasks tail behaviour, and the size of that share, is unknown.
How much weight a median can bear also depends on how many calls it rests on. Muse Spark 1.1’s SEC filing analysis median of 8.2 seconds was measured over 825 synchronous calls [^12], while some of Inkling’s topic figures rest on 25 to 44 calls. How stable those smaller figures would be is not established.
What to do with this
- Before relying on a score, check that the benchmark measured your exact model on your exact task. If it didn’t, measure it yourself.
- Don’t choose a model for a lead that sits inside overlapping intervals. Between tied models, the cheaper one is the sensible default, provided its cost is measured rather than estimated.
- Budget latency on the tail, not the median. Where only a median is published, or it rests on a few dozen calls, measure your own 90th percentile before committing.
Set the quality bar your own workflow requires and inspect the live comparisons at https://fronset.ai/benchmark/.