Price per token vs cost per task across 53 tasks

Every provider publishes a price per million tokens. It is the number most people compare when they choose a model. Our benchmark ranks models differently: by what one run of a task actually costs. That figure covers the tokens the model really used, its reasoning, cache reads, the attempts that failed and were still billed, and what the provider’s invoice adds over its list price (methodology).

So we asked: how well does the price per token predict the cost of a run? We took the snapshot of 16 October 2026 and compared the two for every model on every task, 871 model-and-task pairs across 53 tasks.

The short answer

• Near-equal prices, very different costs. Of 753 model pairs on the same task whose prices are within 25% of each other, 411 (55%) differ by 2× or more in cost per run.
• The widest gap was 59×, between two models on the same task whose prices differ by 11%.
• The price sheet is often off by a lot. Only 501 of the 871 measured costs (58%) are within a factor of two of what the model’s list price predicts for the task. 209 (24%) are off by more than 3×, in one direction or the other.
• One pair in six swaps order. Of 7,961 model pairs on the same task, 1,369 (17.2%) rank one way by price and the other way by cost per run. In those pairs, the model with the cheaper tokens costs a median 1.9× more per run.

Same price sheet, ten times the cost

On bench hypothesis post generation, two models have almost the same list prices:

“Price-implied” is each model’s list price applied to the task’s typical call: 1,548 input tokens and 2,863 output tokens. By that measure Qwen 3.8 Flash is a hair cheaper. Measured per run, it costs 10.5× as much.

Most of the gap is reasoning. Across the 69 calls behind Qwen 3.8 Flash’s figure, it wrote an average of 4,665 output tokens per call, and 4,394 of them were reasoning. That is from our own call records. Priced at list alone, that average call comes to $2.40 per 1,000 runs. That is already 1.5× what its price sheet suggests for this task, and about 4× GPT-6 Luna’s measured cost, which includes what the provider’s invoice adds. The rest of Qwen 3.8 Flash’s figure is the difference between Alibaba’s list price and what we were invoiced on the snapshot date.

GPT-6 Luna’s figure includes its failures: 6 of its 36 attempts failed (30 of 36 succeeded), and the spend on those is in its cost. It is still the cheaper model per run.

To check the numbers on the task page, set the cost mode to “Sync only”. The page opens in “Batch if supported”, and a batch discount is not part of a per-token price.

Same price, different cost: the widest gaps

For each task we took the pair of models whose list prices are closest and whose costs per run are furthest apart. These are the eight widest (sync, per 1,000 runs):

“Gap in list price” compares the two models’ price-implied cost for the task’s typical call. In two of these rows, the model that costs more per run is also the one with the lower list price.

None of these models is cheap or expensive in general. A model that is dear per run on one task is cheap on another. Which one costs less depends on the task.

Why the price sheet misses

A per-token price multiplies by a token count, and the count differs by model and by task:

• Reasoning. A model that thinks before it answers bills those tokens as output, and on some tasks they are most of the output.
• Verbosity. Two models given the same prompt return answers of very different lengths.
• Cache reads. Repeated prompt prefixes are billed at a fraction of the input price, and how much of a prompt hits the cache depends on the provider and on how calls are made.
• Failures. An answer that fails its schema and has to be retried is still billed. We put that spend on the model that produced it.
• The invoice. What a provider bills can differ from its list price. Our figures follow what we are actually invoiced.

None of these appears on a price sheet, and all of them change from one task to the next.

What this does not show

• It is cost on a workload like ours. Cache hits and call sizes follow our prompts and our call pattern. Yours will differ. The direction of the finding should hold, but not every multiple.
• Sync only. Batch discounts differ by provider, and some models have none. Comparing batch costs would mix in a second effect that per-token prices do not show either, so we left it out.
• We left out 35 of the 906 model-and-task pairs. In 10 the cost is modelled rather than measured. In 25 the cost rests on fewer than 25 calls.
• The price-implied figure is a simplification. It uses each task’s average call shape and the base list price (the higher price for long inputs is applied where the typical call needs it). It leaves out cached-input prices and time-of-day pricing.

How we counted

For each task we took every model with a measured cost from at least 25 calls. For each model, we worked out its price-implied cost: its list input and output prices, applied to the task’s typical input and output token counts. We compared that figure with the measured sync cost per run, model by model and pair by pair. Every model named in the tables and the example also gives answers good enough to use on that task, by the benchmark’s own grading, and its cost comes from recent calls.

Ranking by plain output price, by plain input price, or by a 3:1 input-to-output blend instead gives the same picture: between 14.6% and 18.3% of pairs swap order.

Cost per run is the number our router ranks on. How the router picks a model.

We publish these numbers and we sell the API that routes on them. How we keep that honest.