Methodology

What this site does

Until September 2026 this benchmark was published as llm-bench. The name changed; the methodology below, the rubrics and the snapshot history did not.

Most production LLM tasks have a quality threshold, not a quality maximum. Once a model clears the bar, additional capability is unused — you’re paying for headroom your pipeline doesn’t exercise. This site shows you which models clear your quality bar on each step of your pipeline, and ranks them by cost ascending so you can pick the right-sized model rather than reaching for the best-performing model by default.

Quality Score (the metric)

Every model on every task gets a quality score on a 0–10 scale (the internal judge rubric). On the absolute rail it comes from LLM-judge verdicts aggregated via confidence-interval (CI) gates and normalized per-judge offset so a stricter judge doesn’t sink models it rated. Scores are displayed as “X.X / 10” on individual task pages.

Confidence (how much to trust the score)

A quality score of 8.4 isn’t worth much if it could just as easily be 7.0 or 9.8 — same number, very different meaning. Confidence is our answer to “how sure are we?”, derived from how many of the model’s outputs on this task have been graded and how closely those grades agree with each other.

A polling analogy

Think of each graded output as a survey response. The model’s true quality is what an unlimited number of graded outputs would converge on; what we see is a sample. More responses + tighter agreement = narrower margin of error. On a 0–10 scale:

• RANKED — like a major-network political poll the day before an election: thousands of responses, the result holds to within ±2 points on a 100-point scale. We’re as sure as we can practically get. (ci_half ≤ 0.20, i.e. ±0.20 on the 0–10 scale.)
• HIGH — like a restaurant rated by several hundred guests with pretty consistent reviews. Within ±3% of the truth. Order-of-finish vs similar models is stable. (ci_half ≤ 0.30, ±0.30 on the 0–10 scale.)
• MEDIUM — like a small book with thirty reviews that mostly agree. Tight enough to publish — and tight enough that the qualification decision (does it clear the bar?) usually doesn’t depend on the uncertainty — but treat fine-grained rankings of two MEDIUM models with a small grain of salt. Within ±5%. (ci_half ≤ 0.50, ±0.50 on the 0–10 scale.)
• LOW — like a niche restaurant with three reviews ranging from one star to five. Not enough signal yet — hidden everywhere on the site so it can’t mislead. (ci_half > 0.50.)

How it’s computed

Each cell carries a 95% confidence interval (ci_low, point estimate, ci_high). The discrete level above is set by the half-width of that interval: ci_half = 1.96 · σ / √n, where n is the number of distinct graded calls and σ is how far those calls’ scores spread around their mean. A call graded by several judges counts once, at the average of their strictness-corrected scores. Fewer than five graded calls is always LOW. Lower σ and higher n both tighten the band — that’s why more graded calls with more consistent scores mean higher confidence.

Only MEDIUM-or-better cells appear in the qualifying set, regardless of the slider position. This is the same gate the internal selector uses to admit models to production traffic — the public ranking is the subset of cells the system itself trusts.

The Quality Bar (the slider)

The slider asks: how good does a model need to be to handle this task well? That differs by step and by team: one step needs the very best model, another is fine at 80% of it. A model is good enough when it reaches the slider’s share of the best-performing model’s score on the task, with MEDIUM evidence or better; the leaderboard lists those models by cost, cheapest first. The default is chosen at each snapshot from the savings curve (see “Default slider position” below). The best-performing model stays on every chart as the reference point.

Relative-to-best instead of absolute, because task difficulty varies wildly. On a hard task where the best model scores 6.5, “good enough” might be 5.85; on a saturated easy task, “good enough” might be 9.5. Absolute thresholds would leave hard tasks empty and flood easy tasks. The slider position means “right-sized for this task” everywhere.

Cost (the rank key)

Within the qualifying set, rows sort ascending by observed, all-in cost per task run (displayed per 1,000 runs — single runs are sub-cent). This is not a per-token list price: it is what running the task on that model actually costs on our production workload. Each model’s cost comes from its own measured usage on the task — averaged over its recent production calls — including output verbosity, thinking/reasoning tokens, cache reads and writes, server-tool searches, and the spend on its billed failures (a model that wastes money on rejected outputs pays for its own waste). That usage shape is priced at current list rates, band-selected by the real call size, and scaled by the actual-billed-vs-list overhead we reconcile against provider invoices.

Two consequences worth being explicit about. First, the figure is not comparable to providers’ advertised $/1M rates — a model with cheap tokens that thinks at length, or answers verbosely, can cost more per run than a pricier-per-token model that answers tersely. Measuring per run is the point. Second, cache-hit rates depend on our prompt architecture and call cadence, so the number is “cost on a workload like ours” — we publish each task’s typical call shape alongside for context. Where a model has too little recent traffic on a task to measure, we fall back to a modelled estimate at the task’s typical call shape and flag it (cost_basis in the published data: observed, observed_repriced for a stale-but-real shape repriced at current rates, or modelled).

Every model row also shows its cost relative to the cheapest qualifier — the cheapest model that clears the quality bar on that task. That cheapest good-enough model is the reference (marked ★); every other model is shown as a multiple of it: 16x means it costs 16× the cheapest qualifier for no quality benefit above the bar. The cheapest model good enough for the job is the baseline; everything else is what you’d overpay.

We anchor to the cheapest qualifier — not the best-performing model — for two reasons. It answers the question the site is built around (what does the cheapest good-enough model cost, and how much more does everything else?), and it spreads the multiples across a readable range instead of bunching them near one extreme. The reference moves with the slider: raise the bar and fewer — usually pricier — models qualify, so the cheapest-qualifier baseline shifts with it. The best-performing model still appears on every chart (marked best) as the quality reference; it just no longer sets the cost baseline.

Batch vs sync pricing

The cost mode toggle switches the price each model is shown at. Sync only prices every model at standard synchronous rates. Batch if supported uses batch pricing wherever the provider offers it; a model without batch support shows its sync figure in batch mode — running it inside a batch-when-possible pipeline simply costs its sync price, so every model stays in scope and comparable. Both figures are reconciliation-adjusted: the observed usage shape priced at that tier’s list rates, scaled by the actual-billed-vs-list overhead we measure for each provider and model. So the batch-vs-sync gap you see is the discount we actually observe when reconciling provider bills for that model — a per-provider, per-model figure that can sit above or below a provider’s nominal headline batch rate, not a flat list-price discount.

Default slider position — chosen by the data

The default slider position isn’t picked by us; it’s computed at each snapshot via knee detection on the savings curve. For each candidate threshold (75, 80, 85, 90, 95, 100), we compute total pipeline savings at that threshold; the curve’s elbow — where loosening further yields diminishing returns — becomes the default. Clamped to [80, 95] to avoid extremes.

When a new best-performing model lands and the curve reshapes, the default reshapes with it. No human-tuned defaults; just the threshold where the biggest savings live at this snapshot.

Excluded data

• Cells flagged excluded_from_stats (account suspension, quota exhaustion, infrastructure failures).
• LOW-confidence cells (never qualify, never set anchors).
• Cells with fewer graded outputs than the task’s threshold for a settled measurement.
• Deprecated models.
• Meta-evaluation capabilities (judges scoring judges).

Judge panel

The table below lists the current judge panel — the LLMs whose verdicts produce quality scores — by name, provider and role. Only the current panel is shown; retired judges live in the snapshot record but aren’t published.

How a verdict is produced

Grades reach a score by two routes, and they differ in how many judges look at one output:

• A sampled production call. By default every judge on the panel grades the output independently, except a judge whose own model produced it — that judge is left out for that call. The call’s datapoint is the average of the remaining judges’ scores.
• A comparison (fan-out). Mainly while a task is still being measured, and occasionally afterwards to re-check it, one live input is sent to several candidate models, and one judge from the panel grades all of their outputs in a single call. That judge can be grading an output from its own model: on this route the rule above does not apply.

Most of the grades behind the published scores come from comparisons, so most individual verdicts are one judge’s. A published score pools many such verdicts across many calls, from different judges on the panel.

Before verdicts are averaged, each is corrected for its judge’s strictness: for every judge and task we measure how far that judge’s average score sits from the panel’s average on the same task, and subtract the difference. The correction is recomputed daily. It removes a judge’s overall leniency or harshness on a task; it does not tell a strict judge apart from one that happened to grade weaker outputs. A corrected verdict can land outside 0–10. Below 0 it is counted as 0. When a task’s best average comes out above 10, every model on that task is scaled by the same factor (10 divided by that average), so the scores stay on the 0–10 scale and the ratios between them do not change.

Judges are also checked nightly: how often each one agrees with the rest of the panel, how often it returns no usable score, and whether its average is drifting. A judge that crosses the severe thresholds is removed from the panel automatically, unless removing it would leave its provider with no judge — then an operator is alerted to replace it by hand.

What the judges can and cannot tell you

There is no human-label calibration. Every score on this site is an LLM-judge verdict, normalized per judge and confidence-gated; none has been calibrated against human labels, and no human baseline exists behind the numbers. Judges can be older than the models they grade — a known limitation of any LLM-judge panel. Self-preference is not currently corrected for: a judge may favour outputs from its own model family.

What limits it, and what does not:

• On a sampled production call, a judge never grades its own model’s output. A judge from the same vendor can: the rule is per model, not per family.
• On a comparison, the one grading judge can be grading its own model. Those verdicts are counted like any other.
• The strictness correction above adjusts a judge’s overall level on a task. It does not correct a preference for one model over another, so it does not correct self-preference.
• A model from a vendor with no judge on the panel is never graded by a judge of its own family.

What the site does do: every score carries its confidence interval, and scores with too little evidence — a LOW band, or fewer graded outputs than the task needs before its numbers settle — are not published at all (see Excluded data). A narrow interval says the judges were consistent; it does not say they were right.

Whose calls are graded. The pool that feeds these rankings includes verdicts produced when customers’ own calls are evaluated on the free and shared_anon tiers of the API, alongside our own production traffic. What pools is the verdict — a score about a model — never the prompt, the output, or the customer’s identity (Terms §8).