Fronset Benchmark

Best Models for Infrastructure & Utility Work

On utility-type tasks in the benchmark, such as language identification and structured output extraction, the top models are often statistically tied on quality while their prices sit far apart on paper. When the score cannot separate them, two other things can: the price, once you check how it was measured, and how often a call returns output you can use.

The leaders tie; their prices are not like for like

Scores are out of 10, and the ± figure beside a score gives its confidence interval, the margin of uncertainty around it. When two intervals overlap, neither model can be called better. On language identification, Claude Opus 4.8 and GPT-5.6 Luna both scored 9.96, at ±0.15 and ±0.07 respectively [^1][^2]. Qwen 3.8 Flash scored 9.93 (±0.15), tied with both [^2].

Prices are another matter. Qwen 3.8 Flash costs $0.0001 per task, measured on its own usage [^2]. GPT-5.6 Luna is at an estimated $0.0002, and Claude Opus 4.8 at an estimated $0.0037 in batch mode, where requests are queued and answered later [^1][^2]. With one figure measured and the others estimated, the gap is less certain than it looks.

Structured output extraction follows the same pattern. Claude Sonnet 5.5 scored 9.88 (±0.16) at $0.03 per task in batch mode, and Claude Opus 5.5 scored 9.93 (±0.16); its sync cost, for live calls, is recorded separately at $0.13 [^3]. The intervals overlap, so cost and serving mode, not quality, separate them.

On structure tasks, the top answers often tie exactly

The benchmark also counts how often the best answers tie outright. On each judged occasion, competing answers were scored; an exact tie means the top answers got the same score. On table-of-contents extraction, that happened on 75.0% of 76 judged occasions [^4][^5].

The share is 68.1% of 47 for profiled document structure extraction [^6] and 65.6% of 61 for assigning sections to topic clusters, where the median gap between the top answers is 0.0 [^7][^8]. On these tasks, a cheaper or faster model is unlikely to lose much on this measure.

Not every utility-type task looks like this. Query validation shows only 11.5% exact ties and a median gap of 0.7 [^9][^10], and the results here are not enough to pick a model for that kind of work.

Look past the score to answer rate and parse failures

A quality score describes answered calls only, so a model’s answer rate, the share of attempts that get an answer, can outweigh it. On structured output extraction, Kimi K3 scored 9.6 but answered on 76% of attempts [^3]. MiniMax M3 scored 9.36 when it answered, but answered on only 58% [^3][^11].

The extraction tie above needs the same check. Claude Opus 5.5’s answer rate on this task is not given here, so its score should not be treated as the expected result of a call until that rate is confirmed.

Output that cannot be parsed still costs money: it took 24.1% of spend on structured output extraction [^3][^11] and 67.5% on generating visual theme configurations [^12][^13]. Whether the model, the prompt, the requested format or the parser causes these failures is not established.

What to do with this

  • Where top scores overlap, choose on price, but confirm estimated prices against live usage first.
  • Compare prices like for like: batch with batch, sync with sync, measured with measured.
  • Before switching to a cheaper model, measure its answer rate and parse success on your own traffic.
  • Where top answers rarely tie, as on query validation, test candidates on your own inputs before choosing.

Use the live benchmark at https://fronset.ai/benchmark/ to set the quality bar your own workload needs.