Fronset vs picking a model by hand
A leaderboard tells you which model is good in general. An evaluation package tells you which model was good enough on your data the last time you ran it. Fronset keeps grading your task and changes the model that serves it when the answer changes. What each costs you, and why a model picked once doesn't stay the right one.
The short version. There are three common ways to choose the model behind a step in your pipeline: read a leaderboard and try a few models, build an evaluation with one of the evaluation packages out there, or name a task and let Fronset choose. The first two give you an answer once. You then hard-code the model, and the answer stays true until a model ships, a price moves or your inputs drift, and nobody re-checks it. Fronset keeps grading the outputs of your task and changes the model that serves each call when the evidence changes.
What a leaderboard measures
A public leaderboard ranks models on someone else’s tasks. It tells you which models are strong in general. It does not tell you which model is good enough on your step, or which of the good-enough ones is the cheapest to run.
Our own benchmark shows how far apart those two questions are. In the snapshot of 24 September 2026 (53 tasks, 30 models, 55,732 graded samples), with the bar at 90% of the best model’s score on each task:
| What we found | Figure (snapshot 2026-09-24) |
|---|---|
| No model clears the bar on every task | the most is 36 of 53 tasks, and that model is the cheapest good-enough one on only 1 1 |
| The cheapest good-enough model differs by task | 14 different models are the cheapest good-enough one on at least one task (16 when batch prices count) 1 |
| Choosing a strong model that clears the bar is not choosing the cheapest one | in one category, the dearest model that clears the bar costs 343 times the cheapest one, on average across the category’s tasks 1 |
| The answer moves | between the snapshots of 15 and 24 September, the model that was cheapest on the most tasks went from 22 tasks to 13 2 |
So a model picked by hand from a ranking is often good enough and often not the best value, and the right pick for one step can be the wrong pick for the next.
Three ways to choose
With a leaderboard or an evaluation package, finding the best-value model is work you do, step by step, and keep redoing. With Fronset it is built into every call, and there is nothing for you to do: each call runs on the cheapest model that clears the task’s quality bar, and Fronset keeps re-making that choice as the evidence and the prices change. No harness to build, no results to read, no redeploy (see also how routing works).
| How you pick | Grades your production traffic | Re-grades when a new model ships | Changes which model serves | What it costs you | |
|---|---|---|---|---|---|
| A leaderboard and spot checks | Read a public ranking; try two or three models on a few inputs | No | When someone remembers to | When you redeploy | Your time to compare models, and again to change the model in your code and redeploy |
| An evaluation package | Datasets, scorers or LLM judges, and experiments comparing models on your data | Many can, on a sample of live traffic | When you, or a schedule you set, run it again | No: you read the results, change your code and redeploy | Free tiers or paid plans from tens to a few hundred dollars a month, the model calls the evaluation makes, your time to build and run the evaluation, and your time to change the model in your code and redeploy |
| Fronset | You don’t. You name the task, and each call runs on the cheapest model that clears its bar, by design | Yes: a sample of calls is judged, and while a task is being measured some calls are compared across models | On our tasks, new models are measured by replaying stored inputs; on a task of your own, new models are offered and you add them | Yes, automatically, on every call; every response names the model | No per-call fee on the first 10,000 calls a month; model usage and everything beyond: see the Pricing Terms. No manual comparing of models, no changes to your code, no redeployments |
Your evals tell you which model was good enough the last time you ran them. We keep grading, and change the model when the answer changes.
What re-checking takes when a model ships
New models arrive every few weeks. With a model picked by hand, each one asks the same questions: is it good enough on this step, is it cheaper per run (not per token: a model with cheap tokens that writes three times as much costs more), and is it worth a deploy? With an evaluation package you re-run the evaluation for each step, then change the model in your code and redeploy. Without one, the honest answer is usually that nobody checks.
With Fronset:
- On our tasks, when a model ships we replay stored inputs against it and grade the outputs, on our keys and not billed to you. The inputs come from our own traffic and from calls on the free plan (Terms §6.5). If the new model clears the bar more cheaply, it starts serving, and the response headers show the change. Bringing a new model in also raises, for a while, the share of calls that are compared and judged (how routing works, §6).
- On a task of your own, the models you chose stay as they are. A model we add later is listed as offered on the task, and you add it if you want it measured. From then on it is measured on your own traffic (how routing works, §6).
- Every response names the model and provider that answered, its list cost, and how much evidence we hold for it on this task (how routing works, §8).
What it costs you with Fronset
| What you pay | |
|---|---|
| Per call | Nothing for the first 10,000 calls a month, on every plan. Then $2.00, $1.50 and $1.00 per 1,000 calls in marginal bands. Once per request, however many models run behind it, and not on a request that fails (Pricing Terms §3) |
| Model usage on our keys | The provider’s price, with no markup. While a task is being measured, that includes the comparison and judge calls. On one of our tasks with a published rate, the rate, which is set from measured cost and aims at no markup (Pricing Terms §1) |
| Model usage on your own keys | Your provider bills you directly for every model call we make for you, comparisons and judges included (Pricing Terms §3.2) |
| Buying credit | 5.5% of each purchase, with a minimum of $0.80 (Pricing Terms §2) |
For a month of real traffic priced both ways, see Nova, our internal customer: one month on Fronset.
Why a model picked once doesn’t stay the right one
A choice made by hand, or by an evaluation you ran last quarter, assumes nothing has moved since. There is seldom such a stable state:
| What changes | What it does to a hand-picked model | What Fronset does |
|---|---|---|
| Models change | Providers update, re-price and retire models; the one you hard-coded can get worse, dearer or disappear | Each call is ranked on current evidence and current list prices, and a model that is retired or stops clearing the bar stops serving |
| New models appear | Every few weeks there is a candidate that may be as good and cheaper, and checking it is work | On our tasks, new models are measured by replaying stored inputs; on a task of your own they are offered for you to add |
| Your task changes | Inputs drift and steps get reworded; last quarter’s test set stops describing your traffic | A sample of your live calls keeps being judged, so the evidence follows your traffic |
| What counts as good changes | You tighten a rubric or add a check, and the old results no longer answer the question | Calls are judged against the rubric in force; on a task of your own you can change it at any time |
| Provider reliability changes | Outages, rate limits and slow periods hit the one model you depend on | A call that fails, or returns output that fails the task’s checks, is retried on another model |
When Fronset is not for you
These are limits of what Fronset does, not reasons to pick by hand:
- You need streaming, tools, multi-turn conversations, images, or your own sampling settings. Fronset runs one turn per call, text in and text or JSON out, and the task owns its settings.
- You must run one exact model version for good. On a task of your own you can pin one model, but then there is no failover, and on our tasks you cannot pin at all.
When Fronset is the better choice
- The step runs often enough that a cheaper good-enough model is worth real money.
- You don’t want to rebuild and re-run an eval each time a model ships, or redeploy to act on it.
- You want every answer checked against the task’s schema, and repaired or retried on another model, before it reaches you.
- You want to see, on every response, which model answered, what it cost at list and how much evidence stands behind it.
Already using a gateway?
A gateway gives you one API to many models. You still choose the model for each call, or write rules that pick a provider by price, speed or uptime and fall back when one fails. The gateway never sees whether the answer was any good, so the cost-benefit choice stays with you.
For the steps in your pipeline, Fronset is a better gateway. It is an OpenAI-compatible endpoint too, but you name a task instead of a model, and each call runs on the best-value model for that task: the cheapest one whose graded outputs clear the bar, re-chosen as models, prices and your traffic change. The model list is no limit: we can add any text model you need, on request. What you get on top is the choice between them, made on every call. For a detailed comparison with one gateway, see Fronset vs OpenRouter.
The figures from our benchmark are from the snapshot of 24 September 2026; each new snapshot replaces them. What we say about evaluation packages comes from their public documentation and pricing pages, checked on 14 October 2026. Prices and features change. If something here is out of date, write to benchmark@fronset.ai and we’ll correct it.
Fronset Benchmark, snapshot 2026-09-24, quality bar 90%. Read 2026-10-14. ↩︎ ↩︎ ↩︎
Fronset Benchmark: changelog, 2026-09-15 → 2026-09-24, best value (sync). Read 2026-10-14. ↩︎