Migrating from OpenAI

The OpenAI-compatible plane is mounted so that an OpenAI SDK works against it unmodified. Three things change in your code: the base URL, the key, and what you put in model. Everything else about the request and response envelope is the same shape you already parse.

This page is honest about the parts that are not drop-in. Read the two sections marked “fails loudly” before you plan the migration — if your code streams or holds multi-turn conversations, that is real work, not a config change.

The 30-second version

Before:

from openai import OpenAI

client = OpenAI(api_key="sk-EXAMPLE-not-a-real-key")

resp = client.chat.completions.create(
    model="gpt-4o-mini",
    messages=[
        {"role": "system", "content": "You extract financial facts from a public filing as JSON."},
        {"role": "user", "content": "Filing text here..."},
    ],
    temperature=0.2,
)
print(resp.choices[0].message.content)

After:

from openai import OpenAI

client = OpenAI(
    api_key="llmb_i_EXAMPLEKEYPLACEHOLDERxxxxxxxxxxxxxxxxxxxxx",
    base_url="https://api.fronset.ai/v1",
)

resp = client.chat.completions.create(
    model="3.filing-facts-extract",        # a TASK, not an LLM
    messages=[
        {"role": "system", "content": "You extract financial facts from a public filing as JSON."},
        {"role": "user", "content": "Filing text here..."},
    ],
    temperature=0.2,                  # accepted and ignored, see below
)
print(resp.choices[0].message.content)

3.filing-facts-extract is your own task’s slug, as its create returned it (section 2 shows the create).

The response is a standard chat.completion object with one choice and finish_reason: "stop", plus one extra top-level llmbench key (section 5). Your existing parsing keeps working — SDKs ignore unknown fields.

1. Base URL and key

The compat plane is mounted at /v1/, so the full path is exactly POST /v1/chat/completions. Set base_url to https://api.fronset.ai/v1 — your dashboard’s Overview page renders the exact value for your account, and the API host is not necessarily the same host as this documentation site.

Keys start with llmb_, then a one-letter scope tag, then a 43-character random secret:

Both use Authorization: Bearer <key>, which is what the SDK’s api_key sets. Scopes are not interchangeable: a management key on /v1/ is 403 key_plane_mismatch, not 401. A missing or unrecognised key is 401 invalid_api_key. Anything that does not start with llmb_ is rejected without a database lookup.

Only a SHA-256 digest of the key is stored. The plaintext is shown exactly once, at issue or rotation; it cannot be recovered afterwards. The dashboard shows a key as its first 7 characters and last 4 (llmb_i_…wxyz) — that hint is not the secret and will not authenticate.

2. model is a task, not an LLM

This is the conceptual change, and it is the one worth understanding before you write any code.

The Terms define a Capability as “a named unit of work you call — the value you put in the model field of an OpenAI-compatible request. It is not a specific model.” You name the job; the Service selects a model to run it, and may try an alternative if the first one fails our format checks or returns nothing at all. So "gpt-4o-mini" has no meaning here as a routing instruction, and the model that actually ran is reported back in a response header rather than chosen by you.

GET /v1/models therefore lists tasks, not LLMs:

for m in client.models.list().data:
    print(m.id, m.owned_by)
# 3.filing-facts-extract   organization
# summarize-long           llmbench

owned_by is llmbench for tasks we ship and organization for your own. created is always 0 — no creation timestamp is carried on this projection, and a fabricated one would be worse than an obvious zero. Inactive tasks are omitted from the list, and GET /v1/models/{id} returns the same 404 for an inactive task as for one that never existed. The list is drained in full, not paginated.

Create the task first, then call the slug it returns

/v1 only calls tasks that exist: ours by their catalogue slug, yours by the slug your create returned. Nothing is created from a call. A model you have not created — gpt-4o-mini, a typo, or the bare name you sent at create — is 404 model_not_found, on the interactive endpoint and on a batch line alike. A task you have paused (is_active: false) gets the same 404, and a model string longer than 128 characters gets the same 404 model_not_found as an unknown name. An empty model is 400 invalid_request_body with param: "model".

A task of your own is created once, on the control plane, with a management key. It defaults to running on your provider credentials (byok), and selection only considers providers you have an active vaulted key for. So vault a key before you create one: with no vaulted provider that can serve it, the create is refused with 422. A call to a task whose keys you have since revoked fails 503 no_eligible_model, and the message names the exact providers and the endpoint to fix it.

Vault a key first, with a management key:

curl -X PUT https://api.fronset.ai/api/inference/v1/vault/providers/anthropic \
  -H "Authorization: Bearer llmb_m_EXAMPLEKEYPLACEHOLDERxxxxxxxxxxxxxxxxxxx" \
  -H "Content-Type: application/json" \
  -d '{"api_key": "sk-ant-EXAMPLE-not-a-real-key"}'

That stores the key, activates the provider for your account, and runs an auth-check ping. GET /api/inference/v1/vault/providers lists what you have (metadata only, never key material); DELETE revokes. Partial coverage is fine — one vaulted provider in the pool is enough to serve.

Then create the task, with the same management key. Send the bare name and what the task is: "templates": "passthrough" (your calls keep sending messages, as they did to OpenAI) and "output_format": "text" (or "json_schema" with a schema). The full create body, with templates of your own, is in the control-plane reference:

curl -X POST https://api.fronset.ai/api/inference/v1/management/capabilities/ \
  -H "Authorization: Bearer llmb_m_EXAMPLEKEYPLACEHOLDERxxxxxxxxxxxxxxxxxxx" \
  -H "Content-Type: application/json" \
  -d '{"slug": "filing-facts-extract", "display_name": "Financial facts from public filings",
       "templates": "passthrough", "output_format": "text"}'

The 201 body’s slug is what you put in model. It carries your account’s number as a prefix, so read it from the response rather than building it:

{"slug": "3.filing-facts-extract", "name": "filing-facts-extract", "namespace": "customer", "pricing_mode": "byok", …}

So the migration order is: register, vault at least one provider key, create your task, then call the slug the create returned.

3. What fails loudly

These return an error rather than being quietly worked around. That is deliberate: a client written against a feature we do not have should break visibly, not appear to work.

Streaming is refused

{
  "error": {
    "message": "Streaming is not supported by this Service. Re-send the request with stream=false. (diagnostic: 0f3c…)",
    "type": "invalid_request_error",
    "param": "stream",
    "code": "streaming_not_supported"
  }
}

HTTP 400, with x-llmbench-error-code: streaming_not_supported and x-llmbench-error-class: router. There is no SSE code path anywhere in the routers, and the same refusal applies to a batch input line — where it becomes that line’s entry in the error file.

Terms of Service section 2 says the same thing contractually: “Neither mode streams, and we do not offer streaming at this time… setting it to true is refused with an error rather than served — in synchronous and in batch mode alike.”

The check runs inside translation, before a model is selected and before any provider call, so a refused streaming request costs no provider spend and produces no call record. It runs after the quota gate, so it does consume one quota unit and one recorded inbound request — exactly like any other 400.

Migration impact: if you stream for perceived latency, you lose that and there is no workaround on this plane. Plan for a single complete response. If you stream to get incremental usage frames, the full usage block is on the completed response instead.

Multi-turn conversations are not supported in v1

messages must be non-empty, contain at most one user message, and every content must be a string.

Multiple system messages are fine; they are joined with a blank line between them.

Migration impact: this is the biggest one for chat-shaped applications. A conversation loop that appends assistant turns to messages will 400 on its second call. You have to fold the history you want the model to see into the single user message yourself.

Parameters that are refused

These are 400 unsupported_parameter when they carry a meaningful value: tools, tool_choice, functions, function_call, logprobs, audio, modalities, web_search_options, and n when it is not 1.

Benign off-values are accepted rather than spuriously rejected — tools: [], tool_choice: "none", logprobs: false, modalities: ["text"] all pass, because a repointed SDK often sends them by default.

4. What is silently ignored — and how you find out

Sampling parameters are accepted and ignored: the task’s own configuration owns generation parameters.

The rule is simple and there is no allowlist to memorise: every parameter you send that is not in the refused set above is accepted and ignored. That covers the sampling knobs you are used to (temperature, top_p, max_tokens, seed, stop, logit_bias, user, …) and equally any undeclared parameter your client happens to add, such as stream_options.

It is not silent in practice: every parameter that was ignored is echoed back, comma-separated, in the x-llmbench-ignored-params response header, and mirrored into the response body. Log that header during a migration and you will see immediately which of your knobs stopped doing anything.

response_format is honoured:

The right-hand column is the one difference from OpenAI worth planning for: on a catalogue task you name the task and the schema comes with it. If your code sets response_format today, drop it when the model is a catalogue slug, and keep it when the task is one you defined on your own keys. The same rule applies line by line in a batch file.

5. What you gain

Response headers

Every 200 carries six headers, always:

And these when applicable: x-llmbench-request-id (the route id — quote it in a support request and use it to look the route up in your dashboard), x-llmbench-ignored-params, x-llmbench-band (LOW/MEDIUM/HIGH/RANKED — the evidence band of the served model on this task), x-llmbench-phase (explore/exploit/continuous), and x-llmbench-switch-in (an estimate of how many more judged calls remain before the task settles onto cost-optimised routing — omitted both when it is not computable and when the answer is zero).

Three quality headers report the bar and whether it was met: x-llmbench-quality-floor, x-llmbench-quality-met (true/false) and x-llmbench-quality-requested (true only if you set the bar rather than inheriting the task’s default). They are omitted together when the block cannot be resolved — absent means “not resolvable”, never “not met”.

Reading them with the OpenAI SDK, here on one of our tasks. A task that runs its own template takes messages=[] and your content as its template’s variables in llmbench.prompt_vars; GET /v1/models/{id} lists them. A task you created as a pass-through task takes your messages as they are, as in section 2.

raw = client.chat.completions.with_raw_response.create(
    model="structured-content-summarization",
    messages=[],
    extra_body={"llmbench": {"prompt_vars": {
        "content_text": filing_text,
        "content_title": "Annual report 2025",
        "content_type": "filing",
        "content_url": "",
        "summary_structure": "## Financial facts\nRevenue, margin and guidance.",
    }}},
    timeout=900,
)
print(raw.headers["x-llmbench-model"], raw.headers["x-llmbench-cost-usd"])
completion = raw.parse()

A call to a task that is still exploring can take much longer than a plain call, so raise the client timeout, and set max_retries=0 on the client: the OpenAI SDKs retry twice by default, and a retried call runs again from the start and its model usage is billed again.

The same values are also mirrored into the response body under a top-level llmbench key, with the x-llmbench- prefix stripped and hyphens kept. The SDK keeps unknown body fields, so you can read them off the parsed object without dropping to raw responses:

completion = client.chat.completions.create(model="3.filing-facts-extract", messages=[...])
print(completion.model_extra["llmbench"])
# {'call-id': '…', 'capability': '3.filing-facts-extract', 'model': 'claude-…',
#  'provider': 'anthropic', 'service-tier': 'standard', 'cost-usd': '0.00042100',
#  'band': 'HIGH', 'phase': 'exploit'}

Note that the standard model field of the response echoes the slug you requested. The model that actually ran is in the headers and the mirror. If you have a dashboard that graphs resp.model, repoint it.

Per-call routing control: extra_body

The OpenAI SDK merges extra_body into the JSON body root, which is where the llmbench extension block lives. It has exactly nine fields and it is strict — an unknown key is a 400, never a silent no-op, so a typo cannot quietly do nothing:

On a task of your own, selection narrows the task’s configured policy for one call. Anything you omit is inherited, and nothing is persisted:

resp = client.chat.completions.create(
    model="3.filing-facts-extract",
    messages=[{"role": "user", "content": "Filing text here..."}],
    extra_body={"llmbench": {
        "selection": {"strategy": "relative", "quality_threshold": 0.9},
        "trace_id": "job-7741",
    }},
)

On one of our tasks we choose the model and decide how it explores, whichever keys serve it, so every selection field but model_group is refused, as is forced_model below: HTTP 422 shared_task_per_call_field, before any model runs; the message names the fields you sent. force_sync stays allowed. To control them, make the task your own: a copy of ours where copies are available, or a task you define.

On a task of your own every field is yours, on your keys or ours. A per-call model_group other than the task’s own is refused with 422 model_group_differs_from_task unless you send create_private_task_for_new_model_group: true, which runs the call on a private task carrying that group. selection is strict too, and an object that is entirely empty or all-null is treated as absent.

The per-call llmbench block has no objective field — sending one is a 400 invalid_request_body with param: "llmbench"; use selection. (objective does exist on the control plane, as a field you set when you create a task, where it picks that task’s default selection policy. It is not a per-call control.) Note also that strategy: "cost" is not the old cheapest, which kept a quality bar — the equivalent of that is strategy: "relative" with a quality_threshold.

To pin one exact LLM on a task of your own, use forced_model, subject to holding a provider credential for it on your keys. The Terms are explicit that pinning removes the fallback: if a pinned model fails, there is no alternative and the call fails.

6. Batch

The Batch API surface is close to OpenAI’s and mostly ports as-is. POST /v1/files (with purpose="batch"), GET /v1/files/{id}, GET /v1/files/{id}/content, DELETE /v1/files/{id}, POST /v1/batches, GET /v1/batches, GET /v1/batches/{id}, POST /v1/batches/{id}/cancel. Ids are OpenAI-shaped (file-…, batch_…), the batch object is the stock shape, and the status vocabulary is the familiar one (validating, failed, in_progress, finalizing, completed, expired, cancelling, cancelled).

Caps: input files are JSONL, at most 100 MB and 10,000 lines. endpoint must be /v1/chat/completions — anything else is a 400. Client metadata is capped at 16 keys, 64-character keys, 512-character values. If an open-batch cap is configured on your account, exceeding it is 429 too_many_open_batches with Retry-After: 60; with no cap configured, none applies.

Four differences worth planning for:

Every line in a batch must route identically. model, llmbench.selection and llmbench.forced_model must agree across the whole file. This is checked during validation, before any line runs — but it is not an exception from batches.create(): the create call returns 200 with status validating, and the batch then moves to status failed with heterogeneous_routing entries naming the line numbers in its error file. Poll the batch or read the error file rather than expecting the SDK call to raise. messages, prompt_vars, custom_id and response_format are free to vary — those are content. You can hoist the shared values to a package-level llmbench block on the create call (model, system, selection, forced_model, response_format) instead of repeating them on every line; a value present at both levels must match.

completion_window is per-attempt, not total elapsed. It accepts Nm/Nh, floored at 15 minutes, capped at 24 hours, defaulting to "24h". Each provider’s batch tier also has our own per-attempt limit of one hour, and an attempt waits for whichever is shorter. When that time passes without an output, the attempt is cancelled and the line is rescheduled to a different model with a fresh budget, excluding the model that timed out. A line can tighten (never widen) its own budget with llmbench.max_wait_seconds.

There is a separate absolute 24-hour backstop. Every batch also carries an expires_at of created + 24 hours, independent of completion_window. Lines still open at that point become error-file lines with code batch_expired. At submit time, a batch whose estimate cannot finish inside that window is refused up front with 400 batch_cannot_complete, and the message names the line count, the lane, the minutes needed and any of your work already queued ahead of it.

Webhooks go only to registered endpoints. One event per active registered webhook endpoint when a batch reaches a terminal state; zero registered endpoints means zero events and no fallback delivery. Register them on the management plane at POST /api/inference/v1/management/webhook-endpoints/. Events are {"object": "event", "id": "evt_…", "type": "batch.<status>", "created": …, "data": <the batch object>} — a pointer, not results — signed with X-Timestamp and X-Signature: sha256=<hex> over "<timestamp>." + raw_body. The Terms state that webhook delivery is best-effort and that you can check status and results through the API at any time regardless; polling GET /v1/batches/{id} is the authoritative path.

Output lines are the familiar chat.completion shape, and each line’s response.body carries an llmbench mirror — but a narrower one than the interactive endpoint’s. It has call-id, capability, model, provider, service-tier and cost-usd, plus band and ignored-params when they are available. phase, switch-in, request-id and the quality block are interactive-only; do not write batch parsing that expects them.

Error lines carry the OpenAI-shaped error either inside response.body.error (when the failure came back from an invocation) or at top level in error (when it was refused before invocation).

7. Errors

The envelope is unchanged: {"error": {"message", "type", "param", "code"}}. type stays inside OpenAI’s recognised set (invalid_request_error, authentication_error, permission_error, rate_limit_error, insufficient_quota, server_error) so SDK string-matching keeps working. The fine-grained vocabulary lives in code, which is ours and does not match OpenAI’s — if you branch on error.code, that is code to update.

Three headers ride errors from the chat-completions path — everything raised at the gateway, everything mapped back from an invocation, and the throttling and quota 429s:

• x-llmbench-diagnostic — a 32-hex id, also appended to the message as (diagnostic: …). Quote it in a support request.
• x-llmbench-error-class — whose fault it is. This is the fastest retry signal you have:

Note the shape of gateway in particular: it is not only admission control. A tools=[...], an n=2 or a second user message all arrive as x-llmbench-error-class: gateway with no Retry-After, and retrying them will never succeed.

• x-llmbench-error-code — the same value as error.code.

Two narrowings worth knowing:

Plain 404s carry the body only. GET /v1/models/{id}, GET /v1/files/{id} and GET /v1/batches/{id} return the OpenAI error body for a miss with no x-llmbench-* headers and no (diagnostic: …) suffix in the message. Do not write error handling that requires the diagnostic id to be present on every non-2xx.

x-llmbench-request-id only appears once a route was opened. A failure that reached invocation carries it. Failures refused before invocation — quota 429, unknown model, malformed body, the streaming refusal — deliberately carry none, because no route was ever opened and a fabricated id would be worse than an absent one.

429s also carry Retry-After in seconds, and OpenAI SDKs auto-retry 429 honouring it, so short waits self-heal without code changes.

8. Cost expectations

Do not assume the per-token economics you are used to carry over unchanged, and read the pricing terms rather than inferring from a single invoice line.

The part that surprises people: whenever a request is charged at cost — while one of our tasks is still establishing which model performs best, and always on a task of your own — every model call in the chain appears on your invoice, per model: alternatives, the evaluation calls that compare them, failed calls, and our retries after a provider error. Once one of our tasks reaches a stable published rate you are billed that rate instead and the exploration is ours to fund. We never put a task of your own on a published rate. Terms section 11 adds the expectation-setting sentence: because evaluation runs candidate models against each other, its share of your provider spend is materially higher than its share of your call count.

Failure billing differs by charging mode, and each mode states its own rule:

9. Coming from OpenRouter

OpenRouter also speaks the OpenAI wire format, so everything above applies. This section covers only what is different about an OpenRouter call. For how the two compare, see Fronset vs OpenRouter.

Cost is the one to re-read. usage.cost is what OpenRouter charged for the call, in credits (per its API reference, read 2026-10-14). Our x-llmbench-cost-usd is narrower: the provider list cost of the one model call that answered. While a task is still being measured, a request can make more model calls than that one, and on a task charged at cost each of them is billed (Pricing Terms §1.2). The whole chain for a request is route_cost_usd in the route detail (Seeing what you actually paid). The per-call fee is charged separately, once per request, and is in neither figure (Pricing Terms §3).

10. Migration checklist

1. Issue an inference-scope key (llmb_i_…) and a management-scope key (llmb_m_…). Store the plaintext at issue time — you cannot get it back.
2. Vault at least one provider key with the management key, then create each task of your own and keep the slug its create returns. An own-keys task created with nothing vaulted is refused.
3. Set base_url to the value your dashboard shows, and api_key to the llmb_i_… key.
4. Grep your codebase for stream=True on this client. Every one of those call sites needs rewriting.
5. Grep for conversation loops that append to messages. One user message, no assistant turns.
6. Grep for tools=, functions=, n= greater than 1, logprobs=True, and image content parts. All refused.
7. Replace hardcoded LLM names in model with task slugs: a catalogue slug, or the slug your create returned.
8. Log x-llmbench-ignored-params for a day. It tells you exactly which of your parameters stopped mattering.
9. Repoint anything that read resp.model to x-llmbench-model or completion.model_extra["llmbench"]["model"].
10. If you branch on error.code, map the new codes — or branch on x-llmbench-error-class instead, which is coarser and more stable.
11. For batch: confirm every line in a file routes identically, poll for the terminal status rather than relying on batches.create() to raise, and register a webhook endpoint if you want notifications.

The interactive wire contract is also discoverable from the server itself: the compat plane is mounted with its OpenAPI schema and docs view, including the stream field’s refusal description.