Quickstart

The API is OpenAI-compatible. You repoint base_url, swap the key, and the request and response shapes stay the same. Three things differ, and all three are visible in the first call you make:

1. model names a task, not an LLM. We pick the model that serves it.
2. You get two kinds of key, and they are not interchangeable.
3. Streaming is not offered. stream: true is refused with an error, not served buffered.

Before you start

You need an inference-scope API key.

Whether you also need a provider key vaulted with us depends on the task. Every task declares one of two access modes. In managed mode we call the provider on our own account. In bring-your-own-key mode the call runs on your credentials, and model selection only considers providers you have vaulted an active key for.

Tasks you create yourself default to bring-your-own-key, so vault a key before you create one. See Vault a provider key.

1. Get a key

Keys look like llmb_ + a one-letter scope tag + _ + a 43-character secret.

Both are sent the same way: Authorization: Bearer <key>.

Your account’s first keys are issued when it is provisioned. A key is shown once, at creation. After that, issue more with a management key:

curl -X POST https://api.fronset.ai/api/inference/v1/management/keys \
  -H "Authorization: Bearer llmb_m_xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx" \
  -H "Content-Type: application/json" \
  -d '{"scope": "inference", "name": "prod-worker"}'

The plaintext comes back in token and is never readable again — we store only a SHA-256 digest of it. Everywhere else you will see just a hint like llmb_i_…wxyz, which is the first 7 characters and the last 4. That hint cannot authenticate anything.

Three behaviours worth knowing now:

• A key on the wrong plane gets 403 key_plane_mismatch, not a 401. Note that 403 is also what a provider-credential problem returns — read x-llmbench-error-code to tell them apart: key_plane_mismatch is the wrong key, provider_key_missing / provider_key_invalid / provider_key_revoked / provider_account_suspended are your vaulted provider key.
• A token that does not begin llmb_ is rejected without a database lookup.
• Revoking your last active management key is refused, because a hash is one-way and that revoke would lock you out of your own account. Inference keys have no such guard — you can revoke every one of them.

Note that the two planes are separate mounts on the same host:

Your dashboard shows the base URL for your own account. If it differs from the above, use the one your dashboard shows — it is the authority.

2. Vault a provider key

A task you create yourself runs on your provider credentials unless you ask otherwise — its pricing mode is byok. Model selection only considers providers you have vaulted an active key for. With no vaulted provider that can serve it, creating such a task is refused with HTTP 422, and a call to one whose keys you have since revoked fails with HTTP 503:

{"error": {
  "message": "no eligible model: '3.my-classifier' runs on your own provider credentials (pricing_mode=byok), and this tenant has no active vaulted key for anthropic, openai. Vault one via PUT /api/inference/v1/vault/providers/{slug}. (diagnostic: …)",
  "type": "server_error", "param": null, "code": "no_eligible_model"}}

That is an easy failure to misread as an outage, so vault first. It takes one call, with your management key:

curl -X PUT https://api.fronset.ai/api/inference/v1/vault/providers/openai \
  -H "Authorization: Bearer llmb_m_xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx" \
  -H "Content-Type: application/json" \
  -d '{"api_key": "sk-EXAMPLE-not-a-real-key", "config": {}}'

This stores the key, enables that provider for your own-key traffic, and runs an auth-check ping against it. GET /api/inference/v1/vault/providers lists what you have — metadata only, never key material. DELETE on the same path revokes.

Partial coverage is fine. One vaulted provider in the pool is enough to serve; you do not need to vault everything.

3. Create your task

A task is created once, on the control plane, with your management key. Send a bare slug (lower case, starting with a letter; letters, digits, - and _, up to 64 characters), a display_name, and what the task is: "templates": "passthrough" (your call sends the messages, as below) and "output_format": "text" (the answer is text, not a JSON schema). The full create body, with templates of your own and a schema, is in the control-plane reference.

The 201 body carries slug. That is the string you put in model: it carries your account’s number as a prefix, for example 3.press-release-claims. Read it from the response rather than building it. The task defaults to byok, so the key you vaulted in step 2 is what serves it.

Python

import httpx   # installed with the openai package

r = httpx.post(
    "https://api.fronset.ai/api/inference/v1/management/capabilities/",
    headers={"Authorization": "Bearer llmb_m_xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx"},
    json={"slug": "press-release-claims", "display_name": "Claims from press releases",
          "templates": "passthrough", "output_format": "text"},
)
r.raise_for_status()
TASK = r.json()["slug"]   # for example "3.press-release-claims"

TypeScript

const res = await fetch("https://api.fronset.ai/api/inference/v1/management/capabilities/", {
  method: "POST",
  headers: {
    Authorization: "Bearer llmb_m_xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx",
    "Content-Type": "application/json",
  },
  body: JSON.stringify({
    slug: "press-release-claims",
    display_name: "Claims from press releases",
    templates: "passthrough",
    output_format: "text",
  }),
});
if (!res.ok) throw new Error(`create refused: ${res.status} ${await res.text()}`);
const task: string = (await res.json()).slug; // for example "3.press-release-claims"

curl

TASK=$(curl -sS -X POST https://api.fronset.ai/api/inference/v1/management/capabilities/ \
  -H "Authorization: Bearer llmb_m_xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx" \
  -H "Content-Type: application/json" \
  -d '{"slug": "press-release-claims", "display_name": "Claims from press releases",
       "templates": "passthrough", "output_format": "text"}' |
  jq -r .slug)
echo "$TASK"    # for example 3.press-release-claims

If the create is refused, the body has no slug and jq prints null; run the same curl without | jq to read the reason.

4. Make the call

Python

from openai import OpenAI

client = OpenAI(
    base_url="https://api.fronset.ai/v1",
    api_key="llmb_i_xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx",
)

resp = client.chat.completions.create(
    model=TASK,   # the slug your create returned
    messages=[{"role": "user", "content": "Acme Corp today reported third-quarter revenue of $4.2 billion, up 12% year over year."}],
)
print(resp.choices[0].message.content)

TypeScript

import OpenAI from "openai";

const client = new OpenAI({
  baseURL: "https://api.fronset.ai/v1",
  apiKey: "llmb_i_xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx",
});

const resp = await client.chat.completions.create({
  model: task, // the slug your create returned
  messages: [{ role: "user", content: "Acme Corp today reported third-quarter revenue of $4.2 billion, up 12% year over year." }],
});
console.log(resp.choices[0].message.content);

curl

jq -n --arg t "$TASK" '{model: $t, messages: [
  {role: "user", content: "Acme Corp today reported third-quarter revenue of $4.2 billion, up 12% year over year."}]}' |
curl -i https://api.fronset.ai/v1/chat/completions \
  -H "Authorization: Bearer llmb_i_xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx" \
  -H "Content-Type: application/json" \
  --data @-

Use -i. Most of what makes this different from a plain OpenAI call is in the response headers.

The task you just created passes your messages through as the prompt. One of our tasks, or a task you create with a template of its own, runs that template instead: send messages=[] and your content as the template’s variables in llmbench.prompt_vars. GET /v1/models/{task} lists the variables and says which are required, and messages sent to such a task are refused with 400 messages_not_allowed.

resp = client.chat.completions.create(
    model="structured-content-summarization",   # one of our tasks
    messages=[],
    extra_body={"llmbench": {"prompt_vars": {
        "content_text": press_release,
        "content_title": "Acme Corp Q3 results",
        "content_type": "press release",
        "content_url": "",
        "summary_structure": "## Figures\nRevenue and growth.",
    }}},
    timeout=900,
)

A call to a task that is still exploring can take much longer than a plain call. Raise the client timeout, and set max_retries=0 on the client (the OpenAI SDKs retry twice by default): a retried call runs again from the start, and its model usage is billed again.

What model means

model is a task slug — a named unit of work you call. It is not a specific model. (The API’s own word for a task is capability; you will see it in route and field names such as /management/capabilities and x-llmbench-capability.) Which LLM runs is our decision, and it can differ between two calls with the same slug: if a model’s output fails our format checks, or a model returns nothing, we try an alternative for that call rather than failing it.

GET /v1/models lists the slugs you can address, not the underlying LLMs:

curl https://api.fronset.ai/v1/models \
  -H "Authorization: Bearer llmb_i_xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx"

{"object": "list", "data": [
  {"id": "3.press-release-claims", "object": "model", "created": 0, "owned_by": "organization"},
  {"id": "atomic-fact-claim-extraction", "object": "model", "created": 0, "owned_by": "llmbench"}
]}

owned_by is llmbench for tasks we ship and organization for your own. created is always 0 — the projection carries no creation timestamp, and we would rather return an obvious zero than a fabricated date. The response is not paginated: the SDK’s models.list() gets the catalog in one call.

model must name a task you can call

model is either a catalogue slug from GET /v1/models, or the slug your create returned. /v1 never creates a task: create it on the control plane first (step 3), then call the slug it returns.

A name you have not created — an LLM name, a typo, or the bare name you sent at create — is 404 model_not_found, on every lane: an interactive call and a batch input line answer alike. A task you have paused (is_active: false) gets the same 404. A model string longer than 128 characters gets the same 404 model_not_found as an unknown name. An empty model is 400 invalid_request_body with param: "model".

To pin one exact LLM instead, send forced_model in the extension block (below). Pinning removes the failover: if that model fails, the call fails.

What comes back

The body is a standard chat.completion with one choice and finish_reason: "stop". Two notes: usage includes prompt_tokens_details.cached_tokens and completion_tokens_details.reasoning_tokens, and the body’s model field echoes the slug you requested — the LLM that actually ran is in the headers.

Every 200 carries these six:

And these when applicable: x-llmbench-request-id (the route id — quote it in support requests), x-llmbench-band (LOW/MEDIUM/HIGH/RANKED — how much evidence we have for this model on this task), x-llmbench-phase (explore/exploit/continuous), and x-llmbench-ignored-params.

The same values are mirrored into the body under a top-level llmbench key, with the x-llmbench- prefix stripped:

"llmbench": {
  "call-id": "…", "capability": "3.press-release-claims",
  "model": "…", "provider": "…",
  "service-tier": "standard", "cost-usd": "0.00042100",
  "band": "MEDIUM", "phase": "exploit"
}

OpenAI SDKs ignore unknown body keys, so this is safe to leave in place.

Every failure from /v1/chat/completions and the batch routes carries three headers: x-llmbench-diagnostic (a 32-hex id, also appended to the error message — quote it in support requests), x-llmbench-error-class (whose problem it is: gateway, router, provider, your_key, your_quota, your_request) and x-llmbench-error-code. The model list and retrieve routes return the error body only. A 429 also carries Retry-After, which a retrying client should respect.

Two things that will surprise a repointed SDK

Streaming is refused, not ignored. stream: true returns:

{"error": {
  "message": "Streaming is not supported by this Service. Re-send the request with stream=false. (diagnostic: …)",
  "type": "invalid_request_error", "param": "stream", "code": "streaming_not_supported"}}

with HTTP 400. This is contractual — Terms of Service section 2 says the same thing. We refuse rather than quietly serving a buffered response, so a client written against a streaming API fails visibly instead of appearing to stream. The refusal happens before any model is selected, so it costs no provider spend.

Sampling parameters are accepted and ignored. The task owns its generation parameters, so temperature, top_p, max_tokens, max_completion_tokens, presence_penalty, frequency_penalty, seed, stop, user and logit_bias are accepted and do nothing. We do not hide this: their names come back in x-llmbench-ignored-params. Features we cannot honour are refused with 400 unsupported_parameter when you actually enable them — tools, tool_choice, functions, function_call, logprobs, audio, modalities, web_search_options, and n when it is not 1. (Benign off-values such as tools: [] or logprobs: false are accepted.)

Also v1-only limits on messages, on a task that takes them: exactly one user message, string content only (no content-parts / vision), and no assistant, tool or function roles. Multiple system messages are fine — they are joined.

The extension block

Everything Fronset-specific goes in a llmbench object at the root of the body. The OpenAI SDK’s extra_body puts it there for you:

resp = client.chat.completions.create(
    model=TASK,
    messages=[{"role": "user", "content": press_release}],
    extra_body={"llmbench": {
        "trace_id": "pr-88213",
        "selection": {"strategy": "relative", "quality_threshold": 0.9},
    }},
)

The block is strict: an unknown key inside it is a 400, never a silent no-op. The nine fields are version, prompt_vars, template, variant_values, selection, forced_model, idempotency_key, trace_id and max_wait_seconds (batch input lines only — on the interactive endpoint it is a 400). selection and forced_model are yours on a task of your own, like the one above; on our tasks they are refused with 422 shared_task_per_call_field.

The per-call block has no objective field — selection is how you steer one call. (objective does exist, as a different field, when you create a task on the control plane: there it picks the task’s default selection policy.) Note that selection.strategy: "cost" is not the old cheapest, which kept a quality bar — the equivalent of that is strategy: "relative" with a quality_threshold.

Where to go next

• API reference — data plane. The full llmbench and selection field lists, every response header, and the Batch API (/v1/files, /v1/batches) including the two things that differ from OpenAI’s: completion_window is a per-attempt budget (capped by our own one-hour limit per attempt) that renews when a line is retried on another model, and batch webhook events only fire to endpoints you have registered — with none registered, poll instead.
• API reference — control plane. Vault, key management, task and schema registration, usage and quota, webhook endpoints. Management key throughout.
• Error reference. The full code vocabulary and what to do per x-llmbench-error-class.
• Lifecycle and exploration cost. What x-llmbench-phase and x-llmbench-band mean, and how a task is charged while we are still establishing which model serves it best. Read this before your first invoice: see also the Pricing Terms section 1.2.
• Migrating from direct OpenAI. What changes, what does not, and the behavioural deltas.

The Terms of Service, Privacy Policy and Pricing Terms govern use of the API. Where anything on this page differs from those, those govern.