“Models aren’t interchangeable. My prompt is tuned for one.”
You tuned it for one model. We adapt it to each model it runs on, and then measure how well each one does. The comparison isn’t between two or three models: it is every model on the system that can do the task (the current benchmark covers 30), or a group you define, such as US-hosted models or US frontier models, on tasks that run on your own keys.
Your prompt works on the model you optimised it for. The question is whether another model does it as well for a fraction of the cost. For a task you define, “as well” means your terms: you set the output schema and the validators, and every call is checked against them. An answer that doesn’t fit counts as a model failure, and the call is retried on another model. Only models whose graded outputs on your task reach the minimum evidence band serve it. On your own-key tasks you can also pin one model.
“It’s a black box. Why did it pick that model?”
Every response names the model and provider that answered and the list cost. Whenever they can be resolved, it also says how much evidence we have for that model on this task and whether the task is still being measured. For catalogue tasks, the quality data behind the choice is on the public benchmark, with the prompts and rubrics. For tasks you define, it’s in each response and in the task’s status through the API.
“Isn’t this OpenRouter’s Auto Router?”
No. Popularity is not the quality of the answer for your process. As of September 2026, their Auto Router sorts a prompt into about 30 task types and ranks models by what OpenRouter users spent on them over the trailing seven days. Their Pareto router picks the cheapest available model within a band of quality scores, but only for coding, and the score is a public benchmark’s (Artificial Analysis’s coding percentile), which may or may not be a good proxy for your task.
We grade the outputs of your task itself. Judges score them against evaluation criteria written for that task, and every output is checked against its validators and return schema. For a task you define, those are your process’s quality criteria, your schema and your validators. Models are then ranked by that evidence, with confidence bands, and by cost per run.
“Your per-call fee costs more than my tokens.”
It can. If your step already runs on the cheapest model that’s good enough for it, we only add cost, so don’t send us that step. The question is whether you know that it does. On our benchmark, at its default quality step, the top-scoring model costs up to 715× the cheapest good-enough one per run on a single task, and the median task saves 94% (snapshot 2026-10-31, default quality step Balanced, a set share of each task’s best score).
We are a quality-assurance layer: acceptable, reliable quality at the lowest cost we can find. If the cheapest good-enough model is unavailable, the call goes to the next good-enough one, slightly dearer, and you still get your answer. We keep testing after a task is measured, and we adjust how many calls are sent for re-grading as the evidence changes. If a model that did well starts doing worse, or a new model gives the same quality for less, we notice, and your calls move with it.
The per-call fee pays for that system. It is a real cost, shared across every customer, so it is far less than building and running the same evaluation and routing for your own pipeline. What it costs is on the pricing page.
“You’ll run five models on my dime while you ‘explore’.”
To find the models that are good enough for a task, their answers have to be compared and graded by judges. Once one of our managed tasks is measured we charge a published rate that aims at cost, with no markup, so there is no margin to pay for that first evaluation: while the task is still being measured, those comparison calls are on your invoice at cost. The Pricing Terms say so and show an example (§1.2, §3.2).
How much of it you carry depends on your plan. On the free and shared_anon plans, every
customer calling one of our managed tasks shares the cost of getting it to maturity: their
grades are pooled, the task is measured once for all of them, and you may pay only a
fraction of what that takes. On private (contact sales) you carry the cost of getting it
there yourself, but the results are yours alone: only you see them, use the task and
benefit from them. On your own keys, your provider bills you for the comparison calls
directly. On a task of your own you can switch comparison off for one call or for the whole
task; on one of ours, make a copy of your own first, where copies are available. Our own
benchmark runs never touch your keys (Pricing Terms §3.2).
“Free means you take my data.”
Free means you share what we learn from your calls with other customers, and everyone gets lower cost and higher quality from it. We use your data to make that happen, and only for that. Concretely, on the free plan we keep the prompts you send word for word and may re-run them against new models, and the quality grades from your calls on our tasks are pooled with other customers’. Pooled grades can’t be withdrawn, which is why you must send no personal data on the free plan.
On shared_anon you share less: your prompt text is never written to the call record,
though your grades on our tasks are still pooled. On private (contact sales) you share nothing, and
your grades stay with your account. Upgrade before your first call, because each call is
stored under the plan in force when it runs.
“LLM judges are biased. They prefer their own family.”
Each score pools verdicts from several judges from different vendors (the methodology lists them). When a single judge gives the verdict, which is the case when models are compared on a live input, that judge is picked at random from the panel. Each judge’s scores are normalised for how strict or lenient it is, every score carries a confidence interval, and scores we’re unsure of stay hidden. The rubrics are public.
“I pin model versions. Silent switching breaks my pipeline.”
Switching models doesn’t break your pipeline; what matters is whether the answer is right for it. That is decided by the return schema, the validations and the evaluation rubric, which every answer has to pass, whichever model wrote it. The model is named in every response, so a switch is never silent. For managed tasks, the output schema, the rubric and the prompts are public on each task’s benchmark page.
If you have special requirements, make the task your own: a copy of ours, where copies are available, with your evaluation criteria, your models or a pinned model. On our keys a task of your own is billed at cost and never on our published rate; on your keys your provider bills you. Or define your own task, and you set essentially everything: templates, output schema, validators, evaluation criteria and the models it runs on. Running one of our tasks on your own keys does not make it yours: on our tasks we choose the models and the judges, whichever keys serve them.
Also expect these
“A middleman holding my provider keys.” Vaulted keys are encrypted under a per-account key held in Azure Key Vault, and the API never returns them. A zero-custody mode passes a key per request without storing it. Or use managed tasks and keep your keys out entirely (data and privacy).
“Why not run evals myself?” You can, but you will spend a lot of time and money rebuilding what we have built: a harness per step, judges, re-running it when models ship, and redeploying. We offer that for a small per-call fee.
“Will you publish my prompts?”
- Free plan: you contribute your prompts to a shared pool, which improves the system for everyone, and you benefit from others doing the same. We keep the prompts you send word for word, may re-run them on new models, and pool the quality grades from your calls on our tasks with other customers’, irreversibly. We may republish templates you defined if they generalise to other customers, but we never publish the content of your prompt variables.
- The OpenAI-compatible endpoint: on a task that takes free text (a pass-through task), your messages are your inputs; on any other task you send prompt variables, which are your inputs, and the template is the task’s (prompt variables).
- A template you register holds instructions, not data: keep personal data out of it, on every plan (DPA §3.4).
- Stored templates on the native endpoint: on the free and
shared_anonplans, we may adopt a template into the shared library and republish it word for word. private(contact sales): templates stay yours.- Keep commercially sensitive templates off the free and
shared_anonplans, and upgrade before your first call.