Fronset Benchmark

About — the Fronset Benchmark

The Fronset Benchmark: the cheapest model that's good enough, per task. Built from production workloads and pooled customer grades; independent of model vendors.

What this is

The Fronset Benchmark — the right-sized LLM benchmark — has one job: for each step of an LLM-backed pipeline, show the cheapest model that’s actually good enough, not the best model by default.

It runs on a premise most leaderboards ignore: production tasks have a quality threshold, not a maximum. Once a model clears the bar, extra capability is headroom you pay for and never use. So instead of ranking models head-to-head in aggregate, we score them per task and rank the qualifiers by cost.

We publish these numbers, and we sell the API that routes on them

Read that before anything else on this site. The Fronset Benchmark is published by Fronset, and Fronset sells an API that picks the model for your call from these same measurements. We are the referee, and we benefit from the verdict. Here is how we keep that honest:

  1. On tasks of your own, the quality bar is yours. On our tasks we set the bar once the task has settled, and publish it together with the rate.
  2. Every response names the model that answered and what its answer cost, at the provider’s list price.
  3. Prompts, rubrics and immutable snapshots are published — and the rankings include pooled grades from free and shared_anon customers' calls (Terms §8).
  4. The per-call fee is identical whichever model answers, and we take no markup on model usage. Pass-through is billed at the provider’s own rate; where a task runs on a published rate of ours, that rate is set from measured cost, aims at no markup, and is corrected in both directions when the measurement says so (Pricing Terms §1.3). So a cheaper model never pays us more.

One claim we make about our position, and only one: this benchmark is independent of model vendors. No model vendor pays us, sponsors us, or gives us credits — none, from anyone. What it is not is separate from the router; that is the conflict stated above, and it is why the method, not the name, has to carry the credibility.

Where the scores come from

Every score is derived from real production traffic — not synthetic test sets, not self-reported model numbers. Two sources feed it: our own financial-analysis pipeline, and quality grades produced when customers’ own calls are evaluated on the free and shared_anon tiers of the API. What enters that pool is the grade — a score about a model. Never the prompt, never the output, never who sent it. The conditions under which a grade pools, and why pooled grades are not withdrawn afterwards, are in Terms §8.

How it works (the short version)

  • When a task is being measured, its input goes to the candidate models in parallel (fan-out). Most production calls are not evaluated; how many are depends on how much evidence a task already has (Terms §11).
  • A panel of LLM judges grades every output; scores are normalized per-judge, so a strict judge doesn’t sink the models it rates.
  • A score is only published once enough judges agree to trust it (confidence-gated). Anything we’re unsure about stays hidden.
  • Among models that clear your quality bar, rows sort by cost ascending.
  • The default “good enough” threshold is recomputed at every snapshot from where the savings actually are.

Full mechanics — quality score, confidence bands, the slider, the cost math, and what the judges can and cannot tell you — live on the methodology page.

Reproducibility

  • The system and user prompt templates for every task are published on its task page (the “Prompt templates” section).
  • Evaluation rubrics are listed on each task page.
  • One curated example input/output pair per task.
  • Snapshots are immutable; the full JSON payload that drives the site is published with each snapshot at snapshot.json. The scores, prompts and rubrics are public; the code that produces them is not.

Prompt + rubric + model + typical input shape is everything you need to reproduce a given score.

Cadence

Snapshots are published when we run the pipeline — the changelog shows the dates — plus when a notable model lands. There is no fixed weekday. The public site shows the most recent five snapshots; full history lives in our backend. A writeup of each snapshot goes out in the journal — subscribe by email.

Who publishes it

KAPUA Labs LLC, doing business as Fronset. Until September 2026 this benchmark was published as llm-bench; the name changed, the method, the rubrics and the snapshot history did not. Questions, corrections, disagreements: benchmark@fronset.ai.