How Fronset picks a model

For engineers who’d rather check than trust. Each claim below links to the documentation page or the section of our terms it comes from. Where this page and the Terms of Service or Pricing Terms differ, those govern.

In one paragraph. You name a task. For every model wired to that task we keep graded evidence: scores from a panel of LLM judges, each with a confidence interval. Each call runs on the cheapest model whose score clears a bar set relative to the best model on that task. The response tells you which model that was and what its answer cost, and how much evidence we hold for it.

1. You name a task, not a model

The value in model is the name of a task: a template that builds the prompt from your variables, an output schema, optional validators, and a pool of candidate models. Some tasks are ours: we define them and measure them, and you can see them on the benchmark. Others are yours: a task you create through the API, or a copy you make of one of ours. A task of your own carries your account’s number at the start of its name (Terms §2.1).

/v1 only calls tasks that exist. A name we don’t know is refused with 404 model_not_found (Lifecycle and what you pay).

2. Every model earns evidence on every task

• A panel of LLM judges from more than one provider grades outputs from 0 to 10 against the task’s rubric. The panel, and the rubric and prompts of every task on the benchmark, are published (Methodology).
• Before scores are averaged, each judge’s score is corrected for how strict that judge is on that task, measured against the rest of the panel.
• The scores give each model a 95% confidence interval on the task. The interval’s width sets the evidence band: LOW (wider than ±0.50, never shown on the benchmark), MEDIUM (±0.50), HIGH (±0.30) and RANKED (±0.20). A band says how sure we are, not how good the model is.
• We monitor the evaluations on our tasks, and when a rubric or the judges’ instructions need more detail or a change, we change them.
• On a task of your own you can write your own rubric, and your calls on that task are judged against it. Its verdicts stay in your account. So that you can improve the rubric, we keep each evaluated exchange (what the judge saw, the answers and the verdicts) for a review window you control, and show it in the console and through the API (Evaluation review, Terms §11).

3. The bar is relative to the best

Good enough means a share of the best evidenced score on that task: 90% on most of our tasks, with a few set higher or lower. Tasks differ too much in difficulty for one absolute score to work: on a hard task where the best model scores 6.5, good enough is 5.85 (Methodology). A model also needs at least MEDIUM evidence (or the task’s own floor, if higher) before it can be chosen this way.

On a task of your own the bar is yours: set the share per call with quality_threshold, or store it on the task (API reference).

4. The cheapest model that clears the bar serves the call

Cheapest means observed all-in cost per run on real traffic: every token the model actually used on that task, reasoning tokens and failed attempts included, priced at current list rates and adjusted by the gap we measure between list price and what providers actually bill (Methodology). It isn’t the price per token. A model with cheap tokens that writes three times as much can cost more per run. A model with too little traffic on a task to measure is costed from an estimate until it has some.

5. A new task is measured first

While a task has thin evidence, some calls also go to other models, and a judge compares the outputs and picks the answer you get. We choose those models by what would improve the comparison most, favouring the ones we know least about and weighing what each costs; a model we already know well, and that can’t clear the bar, is re-checked only rarely. A task with little evidence can run most of its pool in one call. This happens on /v1/chat/completions too, so such a call can take many minutes (What exploration is).

A task is being measured (exploring) while fewer than half of its model groups have reached MEDIUM. After that it matures, routing goes cost-first, and on our tasks a small share of calls is still compared to catch a model drifting (Terms §11).

• x-llmbench-phase reports comparison activity, not maturity: a task with comparison switched off always reads exploit. The authoritative read is maturity.status on the task, which is exploring or cost_optimised, with maturity.countdown_judged_calls next to it. x-llmbench-switch-in estimates the judged calls still needed for the model that answered (Reading it off a response).
• Who pays for the comparisons. One request is one call for our per-call fee, however many models run behind it (Pricing Terms §3). The model usage behind it depends on the task:

On a task of your own you can switch comparison off for one call or for the whole task. On our tasks exploration is ours to set, whichever keys serve them (Controlling exploration).

6. New models are measured without you doing anything

On our tasks, when a model ships we replay stored inputs against it and grade the outputs. The inputs come from our own production traffic and from calls on the free plan: on free we keep the prompts you send word for word and may re-run them on new models, which is why no personal data belongs on it (Terms §6.5). Replays run on our keys and are not billed to you. On shared_anon and private your prompt text is not stored, so it is never replayed.

Bringing a new model into a task that already has evidence also raises, for a while, the share of your calls that get compared and judged; that work is paid for as in the table above (Terms §11). If the new model clears the bar more cheaply, it starts serving, and the response headers show the change.

A task of your own keeps the models it was given. A model we add to the catalogue later is not added for you: the task lists it under offered_models, and you add it if you want it measured (Control plane). From then on it is measured on your own traffic, at your cost while the task explores.

7. Every output is checked before you get it

If a model’s output doesn’t match the task’s schema or validators, or a model returns nothing, we repair the output or try another model for that call, and the failure counts against the model that produced it. If no model produces a valid output, the call fails with an output_validation_failed error: an output that fails the task’s checks is never returned as a success (Errors). Pinning one model with forced_model, which you can do on a task of your own, removes the alternative: if the pinned model fails, the call fails (Terms §2).

8. Every response shows its work

The model, provider and cost come with every successful response. The others are best-effort: a missing one means we couldn’t resolve the value, never that it is zero or “not met”. x-llmbench-switch-in is also omitted when the answer is zero, and x-llmbench-ignored-params appears only when something was ignored. The same values are mirrored into the response body under llmbench (Response headers). For everything behind one request, the route detail in the portal lists every model call and what it cost (Seeing what you actually paid).

Where it can go wrong

• A relative bar isn’t an absolute one. If every model is weak on your task, 90% of the best can still be weak. Your schema and validators are the absolute check. On a task of your own you can also raise the bar, or the evidence floor with min_confidence.
• Judges are models. They can favour outputs that resemble their own. On a sampled call a judge never grades its own model’s output, but in a comparison the one grading judge can, and a judge from the same provider always can. The strictness correction does not remove that. None of the scores is calibrated against human labels automatically (Methodology). What we do instead is read the evaluations and improve the rubrics; on a task of your own, the evaluation review lets you do the same.
• A narrow interval says the judges agreed, not that they were right.
• The evidence has an origin. The measurements on our tasks started with our own production workload (content and financial analysis), and they include pooled grades from calls on the free and shared_anon plans. Grades on a task of your own are never pooled (Terms §8). If your inputs look different, a task of your own gathers its own evidence, at your cost while it is measured.
• Re-checks are sampled. A model that gets worse is caught by the small share of calls that is still compared, not on the next call.
• The model can change between calls. Never silently: every response names it. On a task of your own you can restrict the pool or pin one model.
• On our tasks, the choice is ours. A call to one of our tasks that sets selection fields or forced_model is refused with 422 shared_task_per_call_field, whichever keys serve it. If the bar must be yours, make a task of your own, or a copy of ours where copies are available, and call that (What one of our tasks will not let you set).
• No uptime remedy. The paid plans carry a 99.5% monthly uptime target with no service credits. The free plan has no target (Terms §15).

The conflict, and how we handle it

We publish the Fronset Benchmark and sell the API that routes on it, under one name, so there is no way to mistake the two for separate parties. This is where our incentives point, so that you can check us.

• The per-call fee is the same whichever model answers (Pricing Terms §3). On your own keys it is all we charge you; your provider bills the models directly.
• The 5.5% credit fee grows with what you spend on our keys. A pricier model means more credit bought (Pricing Terms §2). That is why the choice rests on published evidence rather than on our word.
• You pay the provider’s published price for model usage, with no markup. Our fees are the per-call fee and the credit-purchase fee (Pricing Terms §1). Where one of our tasks has matured onto a published rate, that rate is set from what the work costs us and aims at no markup. It is a prediction: if it runs high it comes down, and if it runs low it goes up. Every change is dated and published in the portal and through the API (Pricing Terms §1, §1.3).
• What you can check yourself: the model and its list cost, on every response; the prompts, rubrics and immutable snapshots for every task on the benchmark; the rankings, which include pooled customer grades (Terms §8); the published rates and their history; and on a task of your own, the bar, because it is yours.

Why “Fronset”? The front set is the set of models that no other model beats on both cost and quality for your task. Among the models with enough evidence, the cheapest one that clears your bar is always in it, so that’s where Fronset picks from.