API Reference: Data Plane

The data plane is where you send work. It speaks the OpenAI wire format, so an OpenAI SDK pointed at it works unmodified for the calls described here. The same key also works on our native endpoints, which are plain JSON rather than OpenAI’s format.

from openai import OpenAI

client = OpenAI(
    api_key="llmb_i_EXAMPLE_KEY_NOT_A_REAL_KEY",
    base_url="https://api.fronset.ai/v1",
)

Two things to know before anything else:

1. The model field names a task, not an LLM. A task is a named unit of work. We pick the model that runs it, and we may pick a different one next call. If you want one exact model on a task of your own, pin it with llmbench.forced_model (see the llmbench extension block).
2. Streaming is refused, not ignored. stream: true returns a 400. There is no SSE code path. See Streaming.

Authentication

Authorization: Bearer <key>. Keys start with llmb_, then a one-letter scope tag:

Scopes are not interchangeable. A management key on /v1/chat/completions is a 403 key_plane_mismatch, not a 401 — the key is valid, it is just on the wrong plane. A missing, unknown, revoked or expired key is a 401 invalid_api_key.

The plane publishes its own OpenAPI schema at /v1/openapi.json, with an interactive docs view at /v1/docs.

POST /v1/chat/completions

What is identical to OpenAI

• The request envelope: model, messages, response_format.
• The response body: id, object: "chat.completion", created, model, choices[0].message, usage. There is exactly one choice and finish_reason is always "stop".
• usage includes prompt_tokens_details.cached_tokens and completion_tokens_details.reasoning_tokens.
• The error envelope: {"error": {"message", "type", "param", "code"}}, with type drawn from OpenAI’s own set so SDK string-matching keeps working.
• 429 responses carry Retry-After in seconds, so an SDK’s built-in retry works.

What differs

Messages

Every task has a template. What you send depends on which kind it is; GET /v1/models/{id} says, in llmbench.prompt_mode:

• template — the task runs its own template, rendered on our side from the variables you send. Send messages: [] and put your content in llmbench.prompt_vars, one entry per variable the template declares. Messages sent to such a task are refused with 400 messages_not_allowed, not ignored; a variable the template needs and you did not send is 422 missing_prompt_variable. llmbench.template picks one of the task’s published templates by key; omit it for the task’s default.
• messages — a pass-through task: its template is the generic one, and your messages are the prompt. The rules below apply.

resp = client.chat.completions.create(
    model="structured-content-summarization",
    messages=[],
    extra_body={"llmbench": {"prompt_vars": {
        "content_text": "…", "content_title": "…", "content_type": "article",
        "content_url": "", "summary_structure": "## Key points",
    }}},
    timeout=900,
)

A call to a task that is still exploring can take much longer than a plain call. Raise the client timeout, and set max_retries=0 on the client (the OpenAI SDKs retry twice by default): a retried call runs again from the start, and its model usage is billed again.

On a pass-through task:

• messages must be non-empty and must contain a user message.
• At most one user message. A second one is a 400 unsupported_value (“Only a single user message is supported in v1”).
• assistant, tool and function roles are a 400 unsupported_value (“Multi-turn conversations are not supported in v1”).
• Every content must be a string. Content-parts arrays (the vision shape) are a 400 invalid_request_body.
• Multiple system messages are allowed; they are joined with a blank line between them.

Parameters we ignore, and parameters we refuse

Most top-level keys that are not model, messages, response_format, stream or llmbench are accepted, have no effect, and are listed back to you in x-llmbench-ignored-params (comma-separated). That includes the usual suspects:

temperature, top_p, max_tokens, max_completion_tokens, presence_penalty,
frequency_penalty, seed, stop, user, logit_bias, stream_options

These are refused with a 400 unsupported_parameter when they carry a meaningful value: tools, tool_choice, functions, function_call, logprobs, audio, modalities, web_search_options, and n when it is not 1.

Benign off-values are accepted rather than spuriously rejected, so a repointed OpenAI client that always sends tools: [], tool_choice: "none", logprobs: false or modalities: ["text"] keeps working. Those params are the exception to the reporting rule above: a benign off-value on one of them — and n: 1 — is accepted silently and is not listed in x-llmbench-ignored-params.

response_format

On a catalogue task the schema is the task’s, and you do not pass one. A managed task is priced and measured against one output shape; a per-call json_schema or json_object would change what the task is, so it is refused rather than silently swapped: HTTP 400, type: invalid_request_error, code: byok_only_field, param: "response_format". Omit response_format (or send {"type": "text"}) to get the task’s own schema, or run the same step as a task on your own keys to choose your own. A catalogue task’s schema is readable through the control plane — GET /api/inference/v1/management/capabilities/{slug} returns its output_schema_ref — and every response is validated against it before it reaches you.

A JSON Schema using a construct we do not support returns 400 invalid_request_body with param: "response_format". Any other type value is also a 400.

The llmbench extension block

The OpenAI SDK merges extra_body into the JSON body root, so llmbench arrives as a top-level key. The block is strict: an unknown key is a 400, never a silent no-op.

resp = client.chat.completions.create(
    model="3.news-item-classifier",
    messages=[],
    extra_body={"llmbench": {
        "prompt_vars": {"news_item": "…"},
        "selection": {"strategy": "relative", "quality_threshold": 0.9},
        "trace_id": "job-4417",
    }},
    timeout=900,
)

3.news-item-classifier stands for your own task’s slug, as its create returned it, here a task whose template declares one variable, news_item. A catalogue task is called by its catalogue slug.

There is no objective field in this block. If you have older code sending one, it now returns 400 invalid_request_body with param: "llmbench" rather than being ignored. (The control plane’s task-creation objective is a different field and is unaffected.)

llmbench.selection

On a task of your own, selection narrows the task’s configured policy for one call. Anything you omit is inherited; nothing is persisted. It is strict — an unknown key is a 400.

On one of our tasks every selection field but model_group is refused, whichever keys serve the task, as are forced_model and, on the native plane, suppress_fanout, exclude_models and metadata.force_judge: we choose the model and decide how the task explores. The refusal is HTTP 422 shared_task_per_call_field, before any model runs, and its message names the fields you sent. force_sync stays allowed. To set them, make the task your own: a copy of ours where copies are available, or a task you define.

strategy: "cost" applies no score gate — it is not “cheapest that is still good enough”. For that, send strategy: "relative" with a quality_threshold.

Calling a task on another model group

Every task runs on one model group, all_models included. A call whose selection.model_group differs from the task’s group runs on a private task of yours: a copy of the called task with that group, whose models, comparisons and statistics come from that group’s models only.

• The private task is billed at cost (pass-through) and is measured at your cost while it explores. It is judged by your account’s private judge pool, the judges of every task of your own. Until you change that pool, its judges are copied from ours and may be served outside the task’s model group, so send only non-sensitive data until you have set your pool.
• The response names it: x-llmbench-capability is the private task’s slug and x-llmbench-private-task-created is true on the call that made it, false after. The llmbench body block mirrors both. From then on call the private task’s slug directly, with no group and no switch.
• Refusals: 422 private_copy_unavailable (copies are not available for your account), model_group_fork_refused (the task cannot be copied; copy_refused_reason says why), model_group_no_eligible_model (on your own keys, no model of the group is on a provider you connected), and 409 private_task_paused (your private task for that group is paused).
• A task you defined yourself is copied the same way: the copy carries your templates, your output contract and your per-model settings, and the task you called is the one it is a copy of. It starts with no evidence of its own, unless your task is itself a copy of one of ours.
• A call to an internal evaluation task, and a Playground run, never makes one.

An empty or all-null selection object is treated as absent.

An unhonourable selection block is refused before any provider call, so a bad spec never costs you anything.

Response headers

Six headers are on every 200:

Eight more appear when applicable:

The three quality-* headers are emitted together or not at all. Absent means “not resolvable”, never “not met” — we would rather tell you nothing than tell you a promise was broken when we could not check.

The same values are mirrored into the response body under a top-level llmbench object, with the x-llmbench- prefix stripped and hyphens kept:

{
  "id": "chatcmpl-3f0c…",
  "object": "chat.completion",
  "model": "3.news-item-classifier",
  "choices": [{"index": 0, "finish_reason": "stop",
               "message": {"role": "assistant", "content": "…"}}],
  "usage": {"prompt_tokens": 812, "completion_tokens": 96, "total_tokens": 908,
            "prompt_tokens_details": {"cached_tokens": 0},
            "completion_tokens_details": {"reasoning_tokens": 0}},
  "llmbench": {
    "call-id": "b1f0…", "capability": "3.news-item-classifier",
    "model": "some-model-v2", "provider": "some-provider",
    "service-tier": "standard", "cost-usd": "0.00042100",
    "band": "MEDIUM", "phase": "exploit"
  }
}

What we bill for the call: x-litellm-response-cost

A 200 can also carry x-litellm-response-cost: what we bill you for the call, in US dollars, 8 decimal places. LiteLLM reads it as the call’s cost, and any client can read it the same way. It is not mirrored into the body. x-llmbench-cost-usd is a different figure: the provider’s list cost of the model call that answered.

The per-call fee is the one for your account’s band when the call was made, so inside the first 10,000 calls of a month it is 0 (the per-call fee). We read your month’s call count every few minutes, so a call made just after your account crosses a band can show the fee of the band before it. On a call charged at cost, an evaluation call we make after answering is billed separately and is not in the value. The header is absent when we cannot state the amount when the call answers, for example on an account that does not use prepaid credit. Absent means unknown, never zero. Your invoice is final.

The model that ran can change between calls

If a model’s output fails our format checks, or a model does not return an output at all, we try an alternative model for that call rather than failing it. That is why x-llmbench-model may differ from one call to the next for the same model slug. The one exception is forced_model: with a pin there is no alternative, and the call fails.

Streaming is refused

client.chat.completions.create(model="…", messages=[…], stream=True)

HTTP/1.1 400 Bad Request
x-llmbench-error-code: streaming_not_supported
x-llmbench-error-class: router
x-llmbench-diagnostic: 0a3d…  (32 hex chars)

{"error": {
  "message": "Streaming is not supported by this Service. Re-send the request with stream=false. (diagnostic: 0a3d…)",
  "type": "invalid_request_error",
  "param": "stream",
  "code": "streaming_not_supported"
}}

We refuse rather than quietly ignoring stream, so that a client written against a streaming API fails visibly instead of appearing to stream. The Terms of Service say the same thing contractually: neither synchronous nor batch mode streams.

Practical consequences:

• The check runs during translation, before any model is selected and before any provider call — so a refused streaming request costs no provider spend and produces no call record.
• It runs after the rate/quota gate, so it does consume one quota unit and one recorded inbound request.
• One check serves both lanes. On this endpoint it is the response; on a batch input line it becomes that line’s entry in the error file.

GET /v1/models

Lists the tasks you can put in the model field — not the underlying LLMs. Listing LLMs would hand you ids this plane rejects, which looks like it worked and then fails.

{"object": "list", "data": [
  {"id": "3.news-item-classifier", "object": "model", "created": 0, "owned_by": "organization"},
  {"id": "summarize-long-doc",    "object": "model", "created": 0, "owned_by": "llmbench"}
]}

• owned_by is llmbench for tasks we ship and organization for your own.
• Each entry carries an llmbench object: prompt_mode (template for a task that runs its own template, messages for a pass-through task) and templates, the keys of its published templates, default first. GET /v1/models/{id} adds each template’s declared variables: {"key", "is_default", "variables": [{"name", "required", "description"}]}. No prompt text is returned.
• created is always 0. We do not map a task’s creation time onto this field, and a fabricated one would be worse than an obvious zero.
• The list is fully drained, not paginated — the SDK calls it once and reads data.
• Inactive tasks are omitted. GET /v1/models/{id} returns the same 404 for an inactive task as for one that never existed.

Errors

This section covers /v1. The native endpoints answer in their own shape; see Native errors.

Every error is an OpenAI-shaped envelope. type stays inside OpenAI’s recognized set (invalid_request_error, authentication_error, permission_error, rate_limit_error, insufficient_quota, server_error); the fine-grained vocabulary is in code.

Errors from /v1/chat/completions — and errors raised while validating a file upload or a batch submission — carry three headers:

A 429 also carries Retry-After in seconds. A failure that reached invocation also carries x-llmbench-request-id; failures refused before invocation — a quota 429, an unknown model, a malformed body, the streaming refusal — deliberately carry no request id, because no route was ever opened and inventing one would be worse than omitting it.

GET/DELETE on a file or a batch, and the cancel route, return the same JSON envelope without those headers and without the (diagnostic: …) suffix — that is where file_not_found, batch_not_found and batch_not_cancellable come from on a lookup. Errors raised by POST /v1/files and POST /v1/batches do carry the headers and the suffix — including file_not_found for an unknown input_file_id, batch_cannot_complete and the create-time insufficient_balance 429.

Any provider message echoed to you is first stripped of key-shaped tokens and truncated to 500 characters.

Reading x-llmbench-error-class

Codes emitted before any model runs

Codes mapped from an invocation failure

A no_eligible_model on a task that runs on your own keys usually means you have not vaulted a key for any provider in its pool; the message names the providers and the vault endpoint to use.

Batch API

The Batch API is /v1/files plus /v1/batches, in OpenAI’s shape.

POST   /v1/files                       upload a JSONL input file (purpose="batch")
GET    /v1/files/{file_id}             metadata
GET    /v1/files/{file_id}/content     raw JSONL bytes
DELETE /v1/files/{file_id}             delete the content
POST   /v1/batches                     create
GET    /v1/batches                     list  (after, limit; limit clamped to 1–100)
GET    /v1/batches/{batch_id}          retrieve
POST   /v1/batches/{batch_id}/cancel   cancel (idempotent)

Input files

Every non-empty line must parse as a JSON object at upload time; full per-line structural validation happens after the batch is created. A file with no request lines is rejected.

Each line:

{"custom_id": "row-0001", "method": "POST", "url": "/v1/chat/completions",
 "body": {"model": "3.news-item-classifier",
          "messages": [],
          "llmbench": {"prompt_vars": {"news_item": "…"}, "max_wait_seconds": 3600}}}

• custom_id — required, a non-empty string of at most 256 characters, unique within the file.
• method must be POST; url must equal the batch’s endpoint.
• body is a chat-completions body, with the same rules and the same llmbench block as the interactive endpoint — plus max_wait_seconds, which is valid only here.

Creating a batch

endpoint must be "/v1/chat/completions"; anything else is a 400.

metadata is capped at 16 keys, keys at 64 characters, values at 512 characters.

Your open-batch cap comes from your own configured limit. 0 means unlimited, and an account with no limits row falls back to 5. Exceeding it is a 429 too_many_open_batches with Retry-After: 60.

A successful create returns 200 with status: "validating". The file is parsed after that, asynchronously — so most content problems surface as a batch that moves to failed, not as an error on the create call.

Delta 1 — every line in a batch must route identically

This is our largest departure from OpenAI’s Batch API. A batch is one group: one task, one selection spec, one pin, one template. model, llmbench.selection, llmbench.forced_model and llmbench.template must agree across every line (a line that omits llmbench.template and a line that names the default’s key disagree). Everything else — messages, custom_id, prompt_vars, response_format — is free to vary. A model group other than the task’s, and llmbench.create_private_task_for_new_model_group, are accepted only in the hoisted block on the create call (see the end of this section): a line may repeat the hoisted values, but lines that agree with each other on a group or the switch without a hoist are still refused.

A file where every line repeats the same values is fine, which is what an SDK naturally emits. Only genuine disagreement is refused. That refusal happens during validation, not on the create call: a file whose lines disagree on routing fails the whole batch (status: "failed"), with the offending line numbers in errors.data and in an error file.

The same holds for the messages rule on a task that runs its own template: a file whose lines send messages to such a task, or send empty messages to a pass-through task, fails the whole batch at validation, with those line numbers (param: "messages").

You can hoist the shared values instead of repeating them, via a top-level llmbench block on the create call. This is ours, not OpenAI’s — a hoisted file will not run against real OpenAI — so it is offered and never required.

client.batches.create(
    input_file_id="file-EXAMPLE0000",
    endpoint="/v1/chat/completions",
    completion_window="24h",
    extra_body={"llmbench": {
        "model": "3.news-item-classifier",
        "system": "You classify public news items.",
        "selection": {"strategy": "relative", "quality_threshold": 0.9},
    }},
)

Hoistable keys are exactly model, system, selection, forced_model, template, response_format, create_private_task_for_new_model_group. A malformed hoist block — an unknown key, a typo’d selecton — is the one routing problem that does 400 on the create call itself, rather than failing the batch later: a bad shared value fails once here instead of routing ten thousand calls under the wrong policy.

A value present at both levels must match. A line may never override package routing. A hoisted file and a fully-specified one build byte-identical requests.

A selection.model_group other than the task’s is chosen once per batch, on the create call: hoist model, selection.model_group and create_private_task_for_new_model_group: true together in the llmbench block. The create resolves the group to the private task carrying it (made once, or reused) and every line runs on that task; the batch object’s llmbench block carries routed_task. Lines may still repeat the hoisted model and llmbench values, as an SDK does, and they agree. A refusal is the create’s own error (a hoisted group without the switch is 422 model_group_differs_from_task), and no batch and no private task are made; the same holds when the create is refused for another reason (a missing file, the open-batch limit). A group, or the switch, set on the lines and not hoisted is never resolved: each line naming a group other than its task’s, or setting the switch, is an invalid line with model_group_differs_from_task, and the batch fails at validation. The same holds when you hoist the group but put model only on the lines: without a hoisted model the create cannot resolve the group, so each line whose group differs from its task’s fails validation. A model-group refusal on the create carries error.refusal_details (for example copy_refused_reason, and retry_after_seconds with a Retry-After header when the daily limit on new tasks is reached). The native package form takes the same three fields at its top level.

Delta 2 — completion_window is per-attempt, not total elapsed

completion_window accepts Nm / Nh, floored at 15 minutes, capped at 24 hours, default "24h". Out of range or unparseable is a 400 invalid_request_body with param: "completion_window".

It is a per-attempt budget, and it is not the only one. Each provider’s batch tier also has our own limit per attempt: one hour. An attempt waits for whichever is shorter. If that time passes without an output, we cancel the attempt and reschedule the line to an alternative model with a fresh budget, excluding the model that timed out. That repeats until the line succeeds, every eligible model has been tried, or the batch hits its absolute expiry.

So with the default "24h" window, a line whose provider has not answered within an hour usually moves to the next model, which may be a different provider’s batch tier. Each line’s llmbench.model in the output file says which model answered it.

A line can tighten its own budget with llmbench.max_wait_seconds, clamped to [900, completion_window]. It can never widen it, and it never lifts our one-hour limit.

Delta 3 — the hard 24-hour expires_at

Independently of completion_window, every batch gets an absolute expires_at of created + 24 hours. When that passes, every still-open line becomes an error-file line with code batch_expired and the message “request did not complete before the batch expired”.

At submit time, a batch whose optimistic estimate cannot finish inside that 24 hours is refused up front with 400 batch_cannot_complete. The message names the line count, the lane, the minimum minutes needed, and how much of your own work is already queued ahead of it.

Statuses

validating, failed, in_progress, finalizing, completed, expired, cancelling, cancelled. The terminal set is failed, completed, expired, cancelled.

Cancel is idempotent: cancelling an already-cancelling or cancelled batch returns 200. Cancelling a terminal batch returns 409 batch_not_cancellable.

The batch object

GET /v1/batches/{id} returns the stock OpenAI object — id, object, endpoint, errors, input_file_id, completion_window, status, output_file_id, error_file_id, the per-state timestamps, request_counts.{total,completed,failed}, metadata — plus a non-standard llmbench block when there is something to say (a submission-time estimate for a window we thought was threatened, and how the package actually ran). SDKs ignore it.

Structural per-line validation failures fail the whole batch (status: "failed") and appear both in errors.data — capped at 100 entries, each with a line number — and in an error file.

Result lines

Output-file lines:

{"id": "batch_req_1041", "custom_id": "row-0001",
 "response": {"status_code": 200, "request_id": "b1f0…",
              "body": {"…a normal chat.completion, including its own llmbench mirror…"}},
 "error": null}

The per-line llmbench mirror carries call-id, capability, model, provider, service-tier, cost-usd, and optionally ignored-params and band. service-tier is where the value batch shows up. The mirror carries no phase and no switch-in — a batch client never sees a live response, so this mirror is the only channel those values would have had, and only the ones that are meaningful after the fact are included.

Error-file lines carry the diagnostic in-body, in one of three shapes.

A line that got a status code — whether from an invocation or from a refusal made before one (a malformed line, a refused parameter, an unknown model, streaming) — keeps the response shape. request_id is null when no route was ever opened:

{"id": "batch_req_1042", "custom_id": "row-0002",
 "response": {"status_code": 502, "request_id": "c9aa…",
              "body": {"error": {"message": "…", "type": "server_error",
                                 "param": null, "code": "provider_error"}}},
 "error": null}

{"id": "batch_req_1043", "custom_id": "row-0003",
 "response": {"status_code": 400, "request_id": null,
              "body": {"error": {"message": "…", "type": "invalid_request_error",
                                 "param": "stream", "code": "streaming_not_supported"}}},
 "error": null}

A line the batch’s absolute expiry cut off has no status code, so it uses the bare error shape:

{"id": "batch_req_1044", "custom_id": "row-0004",
 "response": null,
 "error": {"message": "request did not complete before the batch expired",
           "type": "invalid_request_error", "param": null, "code": "batch_expired"}}

A whole-batch structural failure writes a third shape — no line id, no custom_id, and a line key instead of an OpenAI type:

{"id": null, "custom_id": null, "response": null,
 "error": {"code": "invalid_line", "line": 42, "message": "…", "param": null}}

Delta 4 — batch.* webhook events go only to registered endpoints

When a batch reaches a terminal state, we queue one event per active registered webhook endpoint. Zero registered endpoints means zero events. There is no fallback delivery target — if you have not registered one, poll GET /v1/batches/{id}.

Endpoints are registered on the management plane, with a management key:

POST   /api/inference/v1/management/webhook-endpoints/              register
GET    /api/inference/v1/management/webhook-endpoints/              list (never re-shows secrets)
POST   /api/inference/v1/management/webhook-endpoints/{id}/rotate   new secret
DELETE /api/inference/v1/management/webhook-endpoints/{id}          deactivate

The signing secret is returned exactly once, on create and on rotate. It is not recoverable — rotate mints a new one.

The event body:

{"object": "event",
 "id": "evt_9f2c…",
 "type": "batch.completed",
 "created": 1756000000,
 "data": { "…exactly what GET /v1/batches/{id} returns…" }}

So the event carries status and file ids, not results. Fetch the output file yourself.

Deliveries are POSTed with Content-Type: application/json, X-Timestamp: <unix seconds> and X-Signature: sha256=<hex>. Verify by computing HMAC-SHA256 over the bytes "<timestamp>." + <raw body> with your endpoint’s secret, over the raw body, before parsing it.

Webhook delivery is best-effort and polling is authoritative. The Terms say so, and nothing about a batch finishing depends on a delivery succeeding. Build the poll path even if you use webhooks.

Native endpoints

The native endpoints run the same tasks as /v1, with the same inference key, under the same account rate limit, and are billed the same way. They do not use OpenAI’s format: the body is plain JSON, the task goes in capability_slug (the value you would put in model on /v1), and what we report about a call is in the response body rather than in headers. Use them when you want that body (which model answered, whether a judge picked the answer, the quality block), a result sent to your webhook, or a shorter batch file. For anything else /v1 does the same work.

Base URL: https://api.fronset.ai/api/inference/v1/. The OpenAPI schema is at /api/inference/v1/openapi.json, with an interactive view at /api/inference/v1/docs.

POST /api/inference/v1/invocations/                run a task
GET  /api/inference/v1/invocations/{call_id}       read a call back
GET  /api/inference/v1/batches/{batch_id}/status   the provider batch a queued call joined
POST /api/inference/v1/packages                    submit a compact batch file

The invocations router also serves /invocations/with-vision, /invocations/with-search and /invocations/image. This page does not describe them.

POST /api/inference/v1/invocations/

import httpx

api = "https://api.fronset.ai/api/inference/v1"
inf = {"Authorization": "Bearer llmb_i_EXAMPLE_KEY_NOT_A_REAL_KEY"}

r = httpx.post(f"{api}/invocations/", headers=inf, timeout=900, json={
    "capability_slug": "3.news-item-classifier",
    "workflow_id": "feed-88421",
    "prompt_vars": {"news_item": "…"},
})
body = r.json()

The trailing slash on /invocations/ is part of the path.

On one of our tasks, selection, forced_model, exclude_models and suppress_fanout are refused with 422 shared_task_per_call_field, as described under llmbench.selection.

Unknown top-level fields are ignored, not refused. Unlike the llmbench block on /v1, a misspelt field here has no effect and no error, so check names against this table. The schema lists a few more fields (capability_requirements, estimated_input_tokens, max_wait_seconds); they are for our own clients, so leave them unset. It also shows fields we set on every call, such as external_call; what you send for those is overwritten.

The response

A call answered in the response is a 200:

{"status": "success", "call_id": "6d1f…", "request_id": "a03c…",
 "model_id": "some-model-v2", "provider_id": "some-provider", "service_tier": "standard",
 "parsed": {"label": "earnings"},
 "tokens": {"input": 812, "output": 96, "cached_input": 0, "reasoning": 0,
            "search_invocations": 0},
 "cost_usd": 0.000421, "duration_ms": 1840,
 "recovery_path": null, "panel_version": null, "ignored_params": [],
 "quality": {"band": "MEDIUM", "floor": "MEDIUM", "met": true, "requested": false}}

The invocation endpoints set no x-llmbench-* headers, and they do not report the task’s fan-out phase or switch-in.

A call to a task that is still exploring can run several models and a judge, and take much longer than a single call. Use a long client timeout (the example uses 900 seconds) and do not retry automatically: a retried call runs again from the start, and its model usage is billed again.

Native errors

A failed call is a JSON body with status: "failure":

{"status": "failure", "call_id": "unknown", "request_id": null, "model_id": null,
 "error_category": "config_error", "error_code": "missing_prompt_variable",
 "error_message": "…", "retry_exhausted": true}

• The HTTP status is 400 for every failure except a config_error whose error_code has its own status (table below). A provider timeout or a short balance is a 400 here, not a 504 or a 429 as on /v1, so read error_category rather than the status.
• call_id is "unknown", and request_id may be null, when the call was refused before any model ran.
• retry_exhausted is true when we have stopped trying for this call. false means the failure was on our side and the same call may succeed if you send it again shortly.

We may add categories. Treat one you do not know as a failure of that call.

Some responses do not use the failure body:

A 422 can come in either shape. A body with status is a refusal of the call; a body with detail is a request that did not parse.

Reading a call back

GET /api/inference/v1/invocations/{call_id} returns the call record:

{"call_id": "6d1f…", "status": "success", "model_id": "some-model-v2",
 "provider_id": "some-provider", "service_tier": "standard", "error_category": null,
 "parsed": {"label": "earnings"}, "input_tokens": 812, "output_tokens": 96,
 "cost_usd": 0.000421, "duration_ms": 1840, "recovery_path": null,
 "chain_open": false, "chain_depth": 0}

• status is in_flight, success or failure. in_flight means there is no result yet.
• A failed attempt that we are retrying on another model has chain_open: true, and successor_call_id once the retry has a call of its own. Read that one next.
• An unknown id, or another account’s, is a 404 with {"call_id": "…", "status": "not_found"}.
• On shared_anon and private, a call’s output is deleted shortly after it has been delivered (in the response, or by a 2xx from your webhook), and after 72 hours if it never was (Terms §7). The record then has parsed: null and output_purged: true. On free the output is kept.

Results to a webhook

Register an endpoint once, with a management key, at POST /api/inference/v1/management/webhook-endpoints/ (the routes and the signing secret are described under Delta 4). Then name it on the call:

"recipient_binding": {"kind": "webhook", "payload": {"webhook_endpoint_id": 7}}

That is the only shape we accept. Any other binding, or an id that is not one of your active endpoints, is refused with a 400 (error_category: "config_error") before the call runs.

With a binding, and without force_sync:

• If the task, and a model the call tries, can run in a provider batch, the call may be queued, also after an earlier model failed. You get a 202:

  {"status": "queued", "call_id": "6d1f…", "request_id": "a03c…",
   "model_id": "some-model-v2", "batch_id": "b7e2…",
   "expected_completion": "2026-10-14T09:15:00+00:00",
   "recipient_binding": {"kind": "webhook", "payload": {"webhook_endpoint_id": 7}}}
  

  expected_completion is when the batch closes and goes to the provider (empty if unknown), not when the result arrives. The result comes to your webhook.

• Otherwise the call is answered in the response as usual, and the result is also sent to your webhook. Make your receiver idempotent on call_id.

• If a queued attempt fails, we retry it on another model as a new call, so a delivery can carry a different call_id from the one in the 202. A call that fails for good is delivered with status: "failure".

Send force_sync: true to get every result in the response and nothing on the webhook.

A delivery:

{"call_id": "6d1f…", "status": "success", "capability_slug": "3.news-item-classifier",
 "model_id": "some-model-v2", "parsed": {"label": "earnings"},
 "error_category": "", "error_message": "",
 "usage": {"input_tokens": 812, "output_tokens": 96},
 "metadata": {"…": "your metadata"}}

It is signed like a batch.* event: X-Timestamp and X-Signature: sha256=<hex>, the HMAC-SHA256 of "<timestamp>." + <raw body> with your endpoint’s secret. Verify the raw body before parsing it. Answer with a 2xx: we retry an unacknowledged delivery once a minute for 24 hours, then stop. Delivery is best-effort; GET /invocations/{call_id} is authoritative.

The provider batch behind a queued call

GET /api/inference/v1/batches/{batch_id}/status, with the batch_id from the 202:

{"batch_id": "b7e2…", "status": "submitted", "provider_batch_id": "…",
 "item_count": 40, "success_count": 0, "failure_count": 0,
 "submitted_at": "…", "timeout_at": "…", "created_at": "…"}

status is pending, submitted, processing, completed, distributed, failed or cancelled; distributed means the results have been handed to their calls. A provider batch can carry several of your queued calls to the same task and model, so item_count counts all of them. It is not a /v1/batches batch. To follow one call, read the call back.

Compact batch files: POST /api/inference/v1/packages

A package is a /v1/batches batch with a shorter file. The values every line shares go in the request, and each line carries only what varies:

{"prompt_vars": {"news_item": "…"}}
{"custom_id": "row-0002", "prompt_vars": {"news_item": "…"}}

Upload the file with POST /v1/files (purpose="batch"), then:

r = httpx.post(f"{api}/packages", headers=inf, json={
    "input_file_id": "file-EXAMPLE0000",
    "model": "3.news-item-classifier",
    "completion_window": "24h",
})
batch = r.json()   # the same batch object POST /v1/batches returns

Unknown fields in the request body are ignored. The lines are strict:

• A line carries user, custom_id and prompt_vars, and nothing else. Any other key is a 400 naming the line.
• On a task that runs its own template, leave user off and send prompt_vars; each line runs with messages: []. On a pass-through task, user is the message. Every line carries user or none does; a file that mixes the two is refused.
• custom_id is optional, at most 256 characters. A line without one gets item-1, item-2, … in file order.
• We read the whole file on the create call, so a malformed line is a 400 there, not a failed batch later. The limits under Input files apply to the file as we expand it.

After the create it is an ordinary batch: follow it with GET /v1/batches/{id}, cancel it, fetch its output and error files, and receive its batch.* events, all as described above. Refusals on the create call are OpenAI-shaped and carry the diagnostic headers, as on POST /v1/batches.

Rate limits and quotas

Two separate 429s, distinguished by code:

Both carry Retry-After in seconds, so an OpenAI SDK’s built-in retry handles the first one without changes. A third 429 exists on the batch surface — too_many_open_batches, described under Batches — and insufficient_balance is also a 429; neither is a rate limit.

The numbers

Read your own values, and your current counters, at GET /api/inference/v1/quota/. A null limit means unlimited. Your open-batch cap is not in that payload — it is returned by GET /api/inference/v1/batches/limits, because it carries the opposite null convention: there, 0 means unlimited.

qps_inference is one bucket per account, shared by every key you hold and every data-plane route. /v1/chat/completions, /v1/files, /v1/batches, the native /api/inference/v1/invocations, /api/inference/v1/batches and /api/inference/v1/packages, and a run from the console Playground all draw down the same 10 requests per second, because the counter is keyed on the account and not on the key or the route. Two keys on one account share 10 between them, as do one key used from four processes. Issuing a second key isolates a workload’s credentials, not its rate: a noisy key starves the account’s other keys, so size every client against the account’s number together.

The window is fixed, not sliding. The count resets on the wall-clock second, so a burst straddling one boundary can put 20 requests through in a few milliseconds. Size your client against the average, not against the boundary.

Retry-After is the window, not a measured wait — 1 for qps_inference, 3600 for control_mutations_per_hour. On the data plane the counter has usually rolled well before the second is up.

No tier caps your call volume or your spend. Those four columns exist and are published at GET /quota/, and no plan sets them: what decides whether a call is served is your prepaid balance, not a call counter.

A 400 raised inside the gateway — a bad body, a refused parameter, the streaming refusal — still consumes one quota unit and one recorded inbound request; the counter is an abuse control, not the meter. Authentication failures (401 / 403) and the two provider-key-header 400s (credential_mode_conflict, zero_custody_not_enabled) do not.

A note on cost while a task is exploring

Some tasks run more than one model while we establish which performs best. x-llmbench-phase reports the task’s current fan-out phase and x-llmbench-switch-in estimates the judged calls the served model still needs before its band reaches its target (the task’s own countdown is maturity.countdown_judged_calls). Read exploit as “fan-out is not exploring right now” — it is not a statement about which rate the call was billed at.

Whenever a request is charged at cost — while one of our tasks is exploring, and always on a task of your own — every model call in the chain appears on your invoice, per model, including alternatives whose output was not returned to you, the evaluation calls that compare them, calls that failed, and our own retries after a provider error. Once one of our tasks reaches a stable published rate you are billed that rate instead and the exploration is ours to fund. We never put a task of your own on a published rate.

Because evaluation runs candidate models against each other, its share of your provider spend is materially higher than its share of your call count.

The full rules — including the three different failure-billing rules for a published rate, a request charged at cost, and your own keys — are in the pricing terms and the terms of service, which govern.