Questions we expect
The objections a pipeline engineer raises about Fronset, answered plainly — including where we are the wrong choice.
“Models aren’t interchangeable. My prompt is tuned for one.”
That’s why we measure per task instead of assuming. While your task is being measured, calls are compared across a few models and graded. After that, only models whose graded outputs on your task reach the minimum evidence band serve it. Your schema and validator reject anything that doesn’t fit, and a failed validation counts as a model failure, so the call is retried on another model. On your own-key tasks you can restrict the model pool or pin one model.
“It’s a black box. Why did it pick that model?”
Every response names the model and provider that answered and the list cost. Whenever they can be resolved, it also says how much evidence we have for that model on this task and whether the task is still being measured. For catalogue tasks, the quality data behind the choice is on the public benchmark , with the prompts and rubrics. For tasks you define, it’s in each response and in the task’s status through the API.
“Isn’t this OpenRouter’s Auto Router?”
As of September 2026, their Auto Router sorts a prompt into about 30 task types and ranks models by what OpenRouter users spent on them over the trailing seven days. That measures popularity. Their Pareto router does pick the cheapest available model within a band of quality scores, but only for coding, and the score is a public benchmark’s (Artificial Analysis’s coding percentile). We rank by graded outputs of your task, with confidence bands, then by cost per run.
“Your per-call fee costs more than my tokens.”
It can. The first 10,000 calls a month carry no fee on any plan. After that the fee is $0.002 per call for calls 10,001–50,000, $0.0015 for 50,001–250,000 and $0.001 above 250,000. If your step already runs on the cheapest model that’s good enough, we only add cost, so don’t send us that step. We pay off when the saving per run exceeds the fee, plus the 5.5% credit fee on what you spend, plus the comparison calls made while the task is measured — billed by your provider on your own keys, and on your invoice at cost on a managed task. On our benchmark, at a 90% quality bar, the top-scoring model costs up to 127× the cheapest good-enough one per run on a single task, and the median task saves 91% (snapshot 2026-09-15). The first 10,000 calls tell you which case you’re in.
“You’ll run five models on my dime while you ’explore’.”
While a task is still being measured, some calls are compared across a few models and graded by a judge. On your own keys, your provider bills you for all of it. On managed tasks, those calls are on your invoice at cost until the task has a published rate; after that they’re ours. The Pricing Terms say so and show an example (§1.2, §3.2). You can switch comparison off for one call or for the whole task, and our own benchmark runs never touch your keys (Pricing Terms §3.2).
“Free means you take my data.”
Yes, and here is exactly what we take. On the free plan we keep the prompts you send word
for word and may re-run them against new models. The quality grades from your calls are
pooled with other customers’, and pooled grades can’t be withdrawn. That is the price of
free, and it’s why you must send no personal data on it. The shared_anon plan never
writes your prompt text to the call record, though its grades are still pooled. Only
private, sold by conversation, keeps grades to your account. Upgrade before your first
call, because each call is stored under the plan in force when it runs.
“No streaming, no tools, no chat. What’s the use?”
None, for a chatbot or an agent; use a gateway for those. Pipeline steps don’t stream: a document goes in and JSON comes out. That is the part we do.
“You’re one person. What happens when it breaks?”
On the paid plans we target 99.5% monthly uptime, with no service credits on any plan; the free plan has no uptime target.
| Tier | Channel | First response | Uptime | Status page |
|---|---|---|---|---|
| Free | Docs and email, best effort | none | none | public |
| Shared-anon | Next business day | Published 99.5% monthly target | public | |
| Private | Priority email, named contact | 6 hours | Published 99.5% monthly target, plus incident communications and postmortems | public, plus incident subscription |
First-response times are measured during support hours: 08:00–20:00 Pacific Time, Monday to Friday, excluding US public holidays. The status page is hosted separately from our systems and shows its history. Calls fail over across models and providers. If you need a contractual SLA today, we aren’t the right choice yet.
“LLM judges are biased. They prefer their own family.”
Each score pools verdicts from several judges from different vendors (the methodology lists them ); most individual verdicts are one judge’s, because when models are compared on a live input a single judge grades all of them. Each judge is normalised for how strict it is, every score carries a confidence interval, and scores we’re unsure of stay hidden. The rubrics are public.
Two limits, stated on the same page: self-preference is not currently corrected for — a judge may favour outputs from its own model family — and no score has been calibrated against human labels.
“I pin model versions. Silent switching breaks my pipeline.”
The model can change between calls, which is the point, but never silently: it is named in every response. Once your task has evidence, a new model has to clear the bar on it before it serves; while a task is still being measured, models are compared and graded, which is what produces that evidence. Your schema is enforced on every call. On your own-key tasks you can pin one model or restrict the pool; on a managed task pinning is refused. If you need the same model forever, pin it on your own keys, or don’t use a router.
Also expect these
“A middleman holding my provider keys.” Vaulted keys are encrypted under a per-account key held in Azure Key Vault, and the API never returns them. A zero-custody mode passes a key per request without storing it. Or use managed tasks and keep your keys out entirely (data and privacy ).
“Why not run evals myself with Promptfoo or Braintrust?” You can. It means a harness per step, re-running it when models ship, and redeploying.
“Will you publish my prompts?”
- Free plan: we keep the prompts you send word for word, may re-run them on new models, and pool the quality grades from your calls with other customers’, irreversibly.
- The OpenAI-compatible endpoint: your messages are treated as your inputs, not as a stored template (prompt variables ).
- Stored templates on the native endpoint: on the free and
shared_anonplans, we may adopt a template into the shared library and republish it word for word, with no notice or credit. private(sold by conversation): templates stay yours.- Keep commercially sensitive templates off the free and
shared_anonplans, and upgrade before your first call.