Error Reference

Every error on the OpenAI-compatible plane (/v1/) uses the OpenAI error envelope, so an SDK you already have handles it. This page describes the codes that plane returns, what each one means, and what you do next.

The envelope

{
  "error": {
    "message": "Rate limit exceeded for this tenant. Slow down and retry. (diagnostic: 3f9c1a20d4b7481e9c2f5a6b8d0e7142)",
    "type": "rate_limit_error",
    "param": null,
    "code": "tenant_rate_limited"
  }
}

type stays inside the set OpenAI SDKs recognise — invalid_request_error, authentication_error, permission_error, rate_limit_error, insufficient_quota, server_error — so SDK-level branching keeps working. The precise meaning is in code.

code is append-only. New codes get added; existing ones do not change meaning. Write your handling against x-llmbench-error-class (below) so an unrecognised code still lands in the right branch.

Any message we echo from an upstream provider is stripped of key-shaped tokens and truncated to 500 characters before it reaches you.

Diagnostic id vs call id

These are two different headers on two different outcomes. Do not look for one on the other.

The diagnostic id is also appended to error.message as (diagnostic: ...), so it survives a client that logs the body and throws the headers away. Quote it in a support request.

One nuance worth knowing: when a failure got far enough to open a call record, the diagnostic id is that call record’s id — the same value a success would have reported in x-llmbench-call-id. When the request was refused before any model ran (bad key, unknown model, malformed body, streaming, quota), there is no call, and the diagnostic is a freshly minted 32-character hex id instead.

Error headers

x-llmbench-request-id is deliberately absent on anything refused before invocation. No route was opened, so there is no id, and inventing one would be worse than omitting it.

Not every error carries the header triple. The rule: the triple is attached when the error is raised as a gateway refusal or mapped from an invocation. An error body a plain lookup returns directly carries no x-llmbench-* headers and no (diagnostic: ...) suffix, because the body alone is unambiguous. Lookups in that second group include GET /v1/models/{model_id}, GET /v1/files/{id}, GET /v1/files/{id}/content, DELETE /v1/files/{id}, GET /v1/batches/{id}, and POST /v1/batches/{id}/cancel. Everything from POST /v1/chat/completions, POST /v1/files, and POST /v1/batches carries the full triple.

Triage by class

x-llmbench-error-class tells you who has to act, faster than the code does.

OpenAI SDKs already retry 429s honouring Retry-After, so short gateway and provider waits self-heal without you writing anything.

Codes refused before a model runs

None of these cost provider spend.

internal_error belongs to no band

internal_error (500, or 502 for an unexpected deferred result) carries class router and means something failed on our side. It is listed separately because it is not confined to the pre-invocation stage: it comes from the catch-all around the whole handler, so it can fire after a provider call has already completed. Unlike everything in the table above, it does not guarantee that no provider spend was incurred.

Retry. If it persists, quote the diagnostic id.

The stream: true response, exactly

HTTP/1.1 400 Bad Request
x-llmbench-error-code: streaming_not_supported
x-llmbench-error-class: router
x-llmbench-diagnostic: 7b2e04c9a51f4d3ab8c6e9f012345678

{"error": {
  "message": "Streaming is not supported by this Service. Re-send the request with stream=false. (diagnostic: 7b2e04c9a51f4d3ab8c6e9f012345678)",
  "type": "invalid_request_error",
  "param": "stream",
  "code": "streaming_not_supported"
}}

The check runs before a model is selected, so a refused streaming request produces no provider spend and no call record. The same check serves both lanes: on POST /v1/chat/completions it is the response; on a batch input line it becomes that line’s entry in the error file.

Codes from the invocation stage

These come from the invocation stage — either from a model that ran, or from a refusal made once the task had been resolved. Six of them run no model at all, and are marked below.

If a model’s output fails our format checks, or a model returns nothing, we try a different model for that call rather than failing it — unless you pinned one with llmbench.forced_model, in which case there is no alternative and you get the failure directly.

insufficient_balance also occurs at batch create (POST /v1/batches), refusing the whole batch before any line is enrolled. There it carries class gateway rather than your_quota; the code, the status and the remedy are the same. billing_not_ready occurs at batch create too, with the same code, status, class and remedy.

Re-vaulting a key does not fail on a bad key

PUT /api/inference/v1/vault/providers/{slug} re-runs an auth check against the provider, but a rejected key is not an error: the credential is still stored and you still get a 200. Read the response body instead of the status:

• verified is true only when the check passed.
• last_error carries the provider’s own reason when it did not.

GET /api/inference/v1/vault/providers shows the same two fields for every vaulted credential, so you can confirm a fix without re-sending the key.

What no_eligible_model tells you

The message is one of three:

1. A credential message, when the task runs on your own provider credentials and one or more of its providers has no active vaulted key. It names those providers. A revoked key produces this same message as an empty vault, because a revoked credential stops serving.
2. A quality-bar message, when a min_confidence floor is set on the task and exploration is off, so no model has enough evidence to clear it. It names both settings.
3. A generic sentence when neither applies, naming nothing further.

Batch and file codes

batch_expired is a per-line code, not an HTTP status

Every batch carries an absolute expires_at of created + 24 hours, independent of completion_window. When it passes, any still-open line becomes an error-file line:

{"id": "batch_req_412", "custom_id": "row-88", "response": null,
 "error": {"message": "request did not complete before the batch expired",
           "type": "invalid_request_error", "param": null, "code": "batch_expired"}}

You will never see batch_expired as an HTTP response. Read it from the error file named by error_file_id on GET /v1/batches/{id}.

Reading per-line failures

Error-file lines come in two shapes, and the discriminator is whether a status code is known — not when the failure happened.

• A status code is known. The error sits under response: {"status_code": ..., "request_id": ..., "body": {"error": {...}}}, and top-level error is null. This covers per-line refusals such as an unknown model or stream: true on the line (request_id is null there, since no call was opened) as well as failures from an invocation.
• No status code. response is null and the error sits at top-level error. This is batch_expired and whole-batch validation failures.

Every line carries custom_id, which is how you map a failure back to your input line.

Line codes drawn from the invocation and gateway stages use the same vocabulary as the tables above. Validation failures do not — see next.

Batch validation uses its own, smaller vocabulary

A structurally malformed line does not fail only that line. Validation runs over the whole input file, and any failure fails the entire batch: status becomes failed, no line is ever launched, and the error file lists the offending lines with id and custom_id both null.

These entries have a different shape from everything else on this page: they carry a line number and no type field.

Task-name and task-edit codes (control plane)

These come from POST / PATCH /api/inference/v1/management/capabilities and, for the model-group codes, from the model-group and selection-policy routes, in the control plane’s {"error", "code", ...} body. Every one is refused before anything is written.

Notes

Not every non-2xx comes from here. The control plane (/api/inference/v1/) uses a different envelope: ninja’s {"detail": ...} for auth failures, and {"detail", "code", "retry_after"} for 429s. The code vocabulary on this page is the /v1/ plane’s.

A 429 is not always billing. tenant_rate_limited means slow down; tenant_quota_exceeded and insufficient_balance mean stop until something changes. The type field separates them for you — rate_limit_error versus insufficient_quota — which is exactly how OpenAI’s own SDKs branch.

Quality headers are absent, never wrong. On a success, x-llmbench-quality-floor / -met / -requested are omitted together if they cannot be resolved. Absent means “not resolvable”, not “not met”. Do not read a missing header as a failed promise.