Set a quality bar, not a leaderboard: the catalyst-analysis model that clears it for $0.02

If you build on LLM APIs, the most useful number in a benchmark is rarely the top score. It is the cheapest model that lands within a stated distance of that score. On financial catalyst and scenario analysis, the cheapest model that clears a 90%-of-best bar costs less than half of what the leader does per run, and the benchmark’s own head-to-head judging says the field below the top is too tight to call anyway.

Decide what good enough means first

The benchmark’s routing rule is simple. A model qualifies on a task when its quality is within 90% of the best measured model; on some tasks that bar is raised to 95% and on others lowered to 80% [^1][^2], with the bar set per task and stated. Every score behind it is one judge grading one output against a published rubric on a 0–10 scale, and where one model is reported to have beaten another, both answered the same live input and were scored independently [^1][^2].

That framing turns model choice into a threshold question rather than a ranking question. You decide what quality your product needs, and then pay for exactly that.

The catalyst-analysis numbers

On analyzing catalysts and scenarios, Gemini 3.5 Flash scores 8.8 out of 10 (±0.17) and costs $0.05 per task run in sync mode, measured on this model’s own usage and repriced at current rates [^3][^4]. That is the best measured quality on this task point.

Set the bar at 90% of best, and 90% of 8.8 is 7.92. The cheapest measured option at or above that line is GPT-5.4 Nano at 7.94 out of 10 (±0.21) for $0.02 per task run in sync mode [^3][^4]. Thinking Machines Inkling also clears it at 7.99 out of 10 (±0.42) and costs $0.03 per task run in sync mode, measured on this model’s own usage [^3][^4].

The gap between best quality and cheapest qualifier is 0.86 points on quality and $0.03 on quoted per-task price [^3][^4]. That is a material quality step for less than half the sync price, so the choice is a real one, not a rounding error. Just below the line, GLM-5.3 Flash scores 7.83 out of 10 (±0.38) and costs $0.0063 per task run in sync mode [^4], missing the bar by 0.09 points while being the cheapest near-miss, which is a reminder that where you set the bar matters as much as which model tops the table.

Why the scores overstate the gap

Head-to-head judging on the same catalyst work does not resolve the field below the top into a clear winner. The top two answers scored exactly the same in 20.0% of 30 judged occasions, and finished within half a point in 73.3%, with a median gap of 0.2 on a 10-point scale [^3][^4]. When the leader and the runner-up are that often indistinguishable on a single live input, an average difference between models is real but small relative to the spread between individual answers.

There is also a length effect worth knowing about. Judges in that work preferred longer answers, with the answers judges picked longer than the ones they passed over, at 4,745 against 4,177 output tokens [^3][^4]. Longer outputs are also more output tokens, so answer length sits on both sides of the quality-versus-cost trade.

Check the execution mode before comparing prices

Sync and batch prices are not like-for-like, and a bar-based route should be priced in the mode you will actually run. Running Claude Opus 5 on that same catalyst work through batch costs $0.07 per task against $0.15 synchronously, 50% less [^4]. Only 50 of 68 benchmark tasks are batch-eligible [^1][^2], so for the rest any batch-priced comparison would be fiction.

Price on usable output, not tokens billed

One more thing a leaderboard hides is that some of what you pay for never becomes an answer. On a neighbouring task in the same category, producing trading recommendations spent $25.16 and $15.30 of it, 60.8%, bought nothing: the tokens were generated, billed, and could not be parsed [^5][^6]. A cheap model with a high parse-failure rate is not cheap, so measure cost per usable result rather than the quoted per-task price before you commit to a route.

Where to look next

The numbers above are task points from one snapshot, and the catalyst ranking should not be extended to filing extraction, trading recommendations or reference-heavy writing without checking those tasks directly. Check the live numbers at https://llm-bench.kapualabs.com/ before you commit a route, and the per-task pages such as https://fronset.ai/benchmark/task/catalyst-and-scenario-analysis/ for the current scores, intervals and prices. Set the quality bar your system actually needs there and compare batch against sync pricing yourself.