The Cheapest Tie Wins: How to Route Classification Work Without Overpaying

On relevance, classification and matching tasks, the model with the highest point score is usually not separated from the runner-up by more than its confidence interval. Once you treat overlapping intervals as a tie, the choices that actually move your bill are which cheaper model shares the tie, whether the job can run in batch, and whether the model answers at all. This post pulls those findings out of the live benchmark and shows how to act on them.

Ties are the normal result

Every score on the benchmark is an average on a 10-point scale with a 95% interval built from independent judgments of the same live inputs. The benchmark treats overlapping intervals as a tie and lets cost, cost mode and answer rate decide, rather than treating a higher point score alone as a win. That rule matters because on these tasks the top of the table is crowded.

Matching content to a topic taxonomy shows how crowded. The top two answers scored exactly the same in 24.1% of 29 judged occasions and finished within half a point in 72.4%, with a median gap of 0.2 on a 10-point scale [^1]. If you are choosing a model on a tenth of a point, you are choosing on noise.

The same quality for a fraction of the price

Taxonomy matching is also the clearest case of a tie with a real price difference. GPT-5.6 Sol scores 8.32 out of 10 (±0.23) at $0.15 per task run in batch mode, but that cost is estimated because the cell has no qualifying usage history of its own, while GPT-5.6 Terra scores 8.23 out of 10 (±0.24) at $0.08 per task run in batch mode, measured on this model’s own usage and repriced at current rates [^2][^1]. The intervals overlap, so the quality result is a tie, and Terra is the cheaper way to get it.

Identifying language is the extreme version. GPT-5.6 Sol scores 10.0 out of 10 (±0.02) in batch at $0.0011 per task run [^3], while Qwen 3.8 Flash scores 9.93 out of 10 (±0.15) at $0.0001 per task run in sync mode, measured on its own usage [^4][^3]. The quality gap is small and the cost gap is proportionally large, with the caveat that the two quotes are in different cost modes.

Eligibility and policy checks tell the same story with a wider spread. DeepSeek V4 Flash scores 9.22 out of 10 (±0.29) for checking persona policy eligibility at $0.0008 per task run in sync mode [^5][^6], while Claude Opus 5 scores 9.04 out of 10 (±0.44) at $0.05 per task run in sync mode [^6]. Read as a tie, that is the same quality bar cleared at very different costs, with no quality-cost trade-off at the top.

Batch mode is a bigger lever than model choice

Before you switch models, check whether the job can wait. Running GPT-5.5 on taxonomy matching through batch costs $0.09 per task against $0.21 synchronously, 59% less for work that can wait [^2][^1]. Running Gemini 3.5 Flash on the same kind of taxonomy work through batch costs $0.07 per task against $0.12 synchronously, 43% less [^2][^7][^1].

Cost mode often matters more than model choice. So the order of operations is: decide whether the request is deferrable, pick the cost mode, and only then pick the cheapest model that ties for best in that mode. One more check before you commit: the benchmark flags whether a cost is measured on a model’s own usage or estimated, and many of the lowest quoted costs are estimated because the cell has no qualifying usage history of its own.

A high score you only get some of the time

A quality score is measured on the attempts a model actually answers, so it overstates usable quality wherever the answer rate is low. Claude Opus 5 scores 8.91 out of 10 on Topic Cluster Labeling when it answers, but it only answers on 76% of attempts [^8], while Claude Haiku 4.5 scores 8.03 out of 10 on the same task when it answers, but only on 50% of attempts [^9][^8]. Half of those Haiku calls return nothing usable, which the quality score alone does not show.

Timeouts are the other half of this. The benchmark allows a synchronous call 600 seconds to answer, and against that deadline GPT-5.6 Sol misses on 25.6% of its 4,061 attempts; those misses cost nothing because no tokens were metered, and the deadline is the benchmark’s setting rather than the model’s property [^10][^11][^12]. For your own routing, that means pricing on usable output and planning a retry or fallback wherever the answer rate falls short of full coverage.

Where to look next

The routing rule that falls out of all this is short: send each relevance, classification or matching request to the cheapest model that ties for best on that exact task, in the batch or sync mode you need, with a full answer rate. The live benchmark at https://llm-bench.kapualabs.com/ lets you set the quality bar your application needs, compare task-level scores with their intervals, and check batch against synchronous pricing yourself before fixing a production route. The per-task pages linked in the sources below are the place to start.