Prompt Variables and Capabilities

The value you put in model names a task, not an LLM. A task — a capability — is the durable half of a call: the instructions, the output shape, the validity rules, the policy that picks a model. This page is about defining and editing one.

Every task has a template, and there are two ways its prompt reaches a model. Read the next section before anything else on this page.

Two ways to supply a prompt

A template is a pair of prompts, a system prompt and a user prompt, with {placeholders} for your content. What differs between tasks is who writes the prompt text.

On a task that runs its own template your content goes in the variables, and the instructions are the task’s. The same call works on both endpoints, and the same template renders it. Messages sent to such a task on /v1/chat/completions are refused with 400 messages_not_allowed, not ignored: send messages: [] and put your content in llmbench.prompt_vars. GET /v1/models/{id} lists each published template’s key and the variables it declares.

A task can carry several templates. Each has a key, and one is the default. A call that names no template runs the default; llmbench.template on /v1/chat/completions, or template on the native endpoint, selects another by key; a key that names no published template of the task is refused with 422 template_not_published. On one of our managed tasks a call runs our published templates only: prompt names that are not one of them are refused the same way. To run a template of your own, define your own task, or send the change with allow_breaking_change, which makes it on a copy of your own where copies are available (control plane).

On a pass-through task your messages are the prompt. The v1 fold accepts a single user message plus any number of system messages, which are joined with a blank line between them. Four things are refused with 400:

• an assistant, tool or function role — “Multi-turn conversations are not supported in v1.”
• a second user message — “Only a single user message is supported in v1.”
• no user message at all
• a non-string content (vision / content-parts are not supported in v1)

Prompt variables

A template is text with {placeholder} slots. prompt_vars supplies the values.

POST /api/inference/v1/invocations/
Authorization: Bearer llmb_i_EXAMPLE_KEY_NOT_A_REAL_KEY

{
  "capability_slug": "3.news-item-triage",
  "workflow_id": "feed-88421",
  "prompt_vars": {
    "article_text": "Acme Corp raised its full-year revenue guidance to $17 billion on data-centre demand.",
    "sector": "Semiconductors"
  }
}

capability_slug and workflow_id are required. workflow_id is a free-form correlation string of yours; it is recorded with the call. system_prompt_name and user_prompt_name are optional: omit both and the task’s default template runs, or send template with the key of one of its published templates.

schema_json_string is a variable we fill: where a template has a {schema_json_string} placeholder, the service renders the task’s output schema into it. Do not send it; on a managed task a value you send is refused with 400 byok_only_field.

Substitution rules

• The renderer substitutes only {name}, where name is letters, digits and underscores.
• Substitution is single-pass. Braces inside a value are never re-scanned, so a variable containing { or } — JSON, code, another prompt — is safe.
• A placeholder with no matching key is left in place as literal text. (It will not normally get that far; see the next section.)
• A lone {, a lone }, and { } with nothing between the braces, are left untouched.
• {{name}} is not a supported escape. The renderer matches the inner {name} and substitutes it, leaving {value} — the outer braces survive as literal text.

Write {name} and nothing else. The check described below uses the renderer’s own rule, so a variable is required exactly when the renderer would substitute it. That includes the inner {name} of {{name}} and of ${name}: both make name required, and both are substituted, leaving the outer characters as literal text (${foo} becomes $ followed by the value).

Missing and extra variables

Before any model is called, the service compares the placeholders in your two templates against the keys you supplied.

• Missing a required variable fails the call up front with 422, error_category: "config_error", error_code: "missing_prompt_variable", and a message naming the missing variables. Nothing is sent to a provider. The refused call still counts against your call quota.
• Extra keys on a task served on our keys are refused: a key that is neither a placeholder nor one of the template’s declared variables fails the call up front with 422 undeclared_prompt_variable, and the message names the keys. Nothing is sent to a provider, and the refused call counts against your call quota. GET /v1/models/{id} lists each template’s variables. On a task served on your own provider keys, extra keys are ignored.
• Naming a prompt that does not exist under your account fails the same way (prompt_config_invalid).

prompt_vars on the OpenAI-compatible endpoint

On a task that runs its own template, extra_body.llmbench.prompt_vars are its variables, substituted exactly as on the native endpoint, and messages must be empty. On a pass-through task your messages are passed through as a single value and substitution is single-pass, so the values are not substituted into your message text. They do reach the validator context — a declarative validator rule can read a named entry (see citations_preserved below).

Keys beginning llmbench_ are reserved and rejected with 400 invalid_request_body, param: "llmbench.prompt_vars".

Where your content lives, and where it must not

The published Terms draw the line this way: the Template is the reusable instruction text; prompt variables are “where your content lives”.

That distinction is not cosmetic. Under the Terms’ data-grant table (§6.1), prompt variables are never stored on the shared_anon and private tiers, and retained verbatim on free. Template text, prompt versions and registered schemas are retained on every tier, including private, and are not covered by erasure. The Terms therefore prohibit personal data inside a Template on any tier, and any personal data at all on free.

Read §6.2 alongside that table, because it qualifies “never stored” in two ways that matter if you are relying on it:

• Batch is the big one. A call submitted through the Batch interface cannot be served without keeping its rendered request, so a working copy lives on the call record’s operational metadata on every tier including private. It is stripped when the batch finalizes, with a daily sweep and a 72-hour in-flight backstop behind that, plus a hard 24-hour batch expiry. For batch traffic “never stored” means “not stored beyond the operational window” — hours to a couple of days.
• Benchmark sets you upload are exempt and are stored and re-run on every tier, which is what you uploaded them for.

Practically: put the variable, per-call, potentially sensitive part in prompt_vars, never in template text. Never put an API key, password or provider credential in either.

Creating a capability

Management-scope key (llmb_m_…), on the management plane.

POST /api/inference/v1/management/capabilities/
Authorization: Bearer llmb_m_EXAMPLE_KEY_NOT_A_REAL_KEY
Content-Type: application/json

{
  "slug": "news-item-triage",
  "display_name": "News item triage",
  "description": "Classify a public news item and extract the ticker it concerns.",
  "templates": [
    {"key": "default",
     "system": "You triage financial news items. Answer only with the schema.",
     "user": "Sector: {sector}\n\nArticle:\n{article_text}",
     "variables": [
       {"name": "sector", "required": true, "description": "The issuer's sector."},
       {"name": "article_text", "required": true, "description": "The article, as published."}
     ]}
  ],
  "output_format": "json_schema",
  "output_schema": {
    "type": "object",
    "additionalProperties": false,
    "required": ["category", "materiality"],
    "properties": {
      "category":  {"type": "string", "enum": ["earnings", "guidance", "deal", "other"]},
      "materiality": {"type": "integer", "minimum": 1, "maximum": 5},
      "ticker":    {"type": ["string", "null"], "maxLength": 10}
    }
  },
  "objective": "cheapest_good_enough",
  "batch_eligible": true
}

Fields

evaluation_policy_slug and human_evaluation_policy_slug are 422 on this plane — those govern how much evaluation gets bought and when human review fires, and are not customer-settable. There is no output_schema_class_path on this API at all: you register a JSON Schema, never a Python import path.

The response echoes the resolved configuration, including the task’s templates (keys and declared variables), the prompt names the default template was written under (system_prompt_name / user_prompt_name), returned for reference — a native invocation may omit them, and the task’s default template runs — plus the seeded cells (model + provider) and any dropped_models with a reason (unknown_model, not_vaulted, model_inactive, provider_inactive, missing_flag:<requirement>).

Create-time refusals worth knowing

• 422 invalid_template. A template defect: a placeholder not declared, a declared variable with the wrong required, a validator reading a variable no template declares. problems lists every one. Nothing is created.
• 422 output_format_required. No output_format, or a schema that does not match it.
• 422, no eligible model. The task would be created but unable to serve anything — every model you named was dropped. dropped_models says why. Nothing is created; the whole create rolls back.
• 422, BYOK not serviceable. The task runs on your credentials and you hold no active vaulted key for any provider in its pool. The message names the providers and the route to fix it (PUT /api/inference/v1/vault/providers/{slug}). Partial coverage is fine — one usable provider in the pool is enough.
• 422, managed requires a billing account. pricing_mode: "managed" needs a billing customer on the account.
• 404, a named policy slug does not exist.

Prompts

A capability owns exactly two prompt names, derived from its slug:

news-item-triage  ->  NEWS_ITEM_TRIAGE_SYSTEM_PROMPT
                    NEWS_ITEM_TRIAGE_USER_PROMPT

(uppercased, hyphens to underscores). Omit name in the prompts block and you get these; they are what the create/update response reports back.

You may supply your own name — but only if it is not already taken. A name already in use elsewhere in your account, outside this capability’s own namespace, is refused with 409, because writing it would repoint every other task that resolves by that name. The practical consequence: a custom-named prompt can be created once and is not editable through this route afterwards. Omit name unless you have a reason.

Read the live content of a prompt of your own with:

GET /api/inference/v1/management/prompts/{name}

which returns content, content_hash, version_counter, template, is_active. A prompt of our catalogue (the ones our shared tasks run on) answers 404, as an unknown name does.

Editing a prompt

Send prompts again on a PATCH:

PATCH /api/inference/v1/management/capabilities/news-item-triage
{"prompts": {"user": {"content": "Sector: {sector}\n\nArticle text:\n{article_text}"}}}

One rule governs edits: the set of {placeholders} must not change. Renaming or dropping a slot silently decouples the template from its call sites — the slot renders as literal text and the model is asked about something it was never shown — so an edit that changes the placeholder set is refused and that prompt is left as it was. Change the wording freely; to change the variables, change the call site and the template together by creating a new capability, or add the new task under a new slug.

Send prompt edits on their own. Prompts are written before the rest of a patch, and the system prompt before the user prompt — so if a two-prompt edit is refused on the second, the first has already been written.

Editing content that hashes identically to the current content is a no-op. Every render records which prompt version ran, so a call is always traceable to exact instruction text.

PUT /api/inference/v1/management/prompts/{name} is not an edit route. Without a config:direct_edit grant it returns 403; with one it returns 405 and points at the governed release surface. Edit prompts through the capability PATCH instead.

Prompts are normalised per model (optimize_prompts)

With optimize_prompts on (the default), a template is rewritten once per model, per template version and per task, so that it suits the model that will serve it. To do that we send the template text, the task’s description and evaluation criteria and the target model’s name to an adaptation model; your variables are never sent. The rewrite is cached, and it must preserve your {placeholder} set exactly; if it does not, or if it fails for any reason, the original template is used. The first call per model is slower, because the rewrite runs inside it. On a task billed at cost the rewrite’s tokens are part of that call’s bill; on a task that runs on your keys they are not billed to you, and the rewrite runs on our keys. Recorded provenance always references your template, never the rewritten text.

Turn it off with "optimize_prompts": false on a PATCH, on your own tasks only; it cannot be changed on one of ours. With it off the template is sent to every model as written.

Output schemas

A capability can carry a JSON Schema that every output must satisfy. Register it inline at create/update time, or register it separately and reference it.

POST /api/inference/v1/output-schemas/
{"name": "triage-result", "json_schema": { ... }}

Registration is idempotent by content: identical content returns 200 with the existing ref, new content returns 201. The ref looks like schema:triage-result@1.

A capability stores a pinned ref (@version). Registering a new version does not move a task onto it — repointing is an explicit capability update. That is deliberate: your task’s definition of a valid answer should not change under you.

The supported subset

Schemas are compiled into a validating model over a bounded subset. Anything outside it is rejected at registration, with a JSON pointer to the offending node — never at call time.

Supported: root type: "object"; string, integer, number, boolean, object, array; properties; required; description; string enum; ["T", "null"] unions; anyOf (including a {"type": "null"} branch for optionals); internal $ref into $defs / definitions; minLength / maxLength, minimum / maximum, minItems / maxItems; items as a single object.

Rejected: oneOf, allOf, not, if / then / else, patternProperties, dependentSchemas, propertyNames, unevaluatedProperties, contains; external $ref; recursive $ref; tuple-form items; non-string enum; open objects (additionalProperties must be false or absent).

Limits: nesting depth 5, 200 nodes, 256 KB.

A property not listed in required is optional and may come back null. If a field must always be present, list it in required.

Per-call overrides

• Native: output_schema_ref_override on the request body.
• OpenAI-compatible: response_format. {"type": "json_schema"} registers or resolves your schema for that call; {"type": "json_object"} returns raw text checked to be JSON-parseable; {"type": "text"} or absent uses the task’s own schema.

An unsupported construct in response_format returns 400 invalid_request_body with param: "response_format".

Validators

A schema decides whether an output is well-formed. A validator decides whether it is acceptable. Set one per task with default_result_validator at create time, or per call with result_validator_override (native).

A validator that rejects an output makes that attempt a failed attempt: the model is excluded and the call is retried on an alternative model — which is the failover the Terms §2 describe. Only when the retries are exhausted does the call fail, as 500 output_validation_failed on the OpenAI-compatible surface.

Resolution is fail-closed. A non-blank validator name that resolves to nothing fails the call before any provider is contacted rather than silently skipping validation — and the same check runs at write time, so a typo is refused when you set it, not discovered in production.

Built-ins

Name one directly in default_result_validator:

A small number of built-ins are operator-only; naming one of those is refused at write time.

Your own validators

POST /api/inference/v1/validators/
{
  "name": "triage-sane",
  "spec_type": "declarative",
  "spec": {
    "spec_version": 1,
    "mode": "all",
    "rules": [
      {"type": "enum", "field": "category",
       "values": ["earnings", "guidance", "deal", "other"]},
      {"type": "numeric_range", "field": "materiality", "min": 1, "max": 5},
      {"type": "forbidden_substring", "field": "*",
       "values": ["as an AI language model"], "case_insensitive": true},
      {"type": "regex_match", "field": "ticker",
       "pattern": "^[A-Z]{1,5}$", "allow_missing": true}
    ]
  }
}

The spec is compiled and validated when you register it, with a pointer (rules[2]) to any offending rule. Re-posting the same name updates it and bumps its version only when the content changes. POST /api/inference/v1/validators/{name}/deactivate removes it from resolution; GET / and GET /{name} read them back.

Rule types — all rules must pass (mode: "all" is the only mode in v1):

Field addressing is a dot-path through the parsed output’s objects (account.reference); "*" means the whole output serialised as JSON. Only object keys resolve — an element of an array cannot be addressed, so a path segment that is a list index never matches anything. A field that is absent fails the rule with field_missing:<path> unless the rule sets allow_missing: true.

Limits: spec_version must be 1; spec ≤ 32 KB; ≤ 32 rules; regex pattern ≤ 512 characters; the subject is truncated to 256 KB before substring and regex checks.

The named_builtin form wraps a built-in with parameters instead:

{"name": "promo-280", "spec_type": "named_builtin",
 "spec": {"builtin": "promo_char_limit", "params": {"limit": 280}}}

Variant axes

variant_axes stratifies a task’s quality evidence, so a model that is strong on one kind of input is not credited for another. Axes are ordered, and selection walks from the most specific key down to the aggregate, dropping the rightmost axis at each step.

{"variant_axes": ["domain"]}

then per call:

{"variant_values": {"domain": "finance"}}

Resolvers exist for domain, language, input_size_band, agent and tenant. A value you pass in variant_values always wins over the derived one — so an axis name with no resolver only ever has the value you send.

Adding an axis does not create per-variant evidence retroactively; the task keeps serving from the aggregate pool until the more specific level has evidence of its own.

Editing a capability

PATCH /api/inference/v1/management/capabilities/{slug}

Patchable: display_name, description, prompts, output_schema / output_schema_ref, variant_axes, models, objective / selection_policy_slug, rollout_policy_slug, fanout_policy_slug, batch_eligible, idempotency_ttl_seconds, pricing_mode, evaluation_criteria_override, is_active.

default_result_validator is not patchable. A PATCH carrying it returns 400, field not patchable: 'default_result_validator'. Set it at create, or override it per call with result_validator_override on the native endpoint.

Two behaviours to plan for:

A model-set change that would empty the pool is refused (422) with nothing applied. Removals bench a model rather than deleting it, so its history survives and re-adding it later is a one-line patch.

What a task measures is yours on your own tasks and ours on a shared one. On a task you created, the prompts, the judging rubric (evaluation_criteria_override), the output schema, the validator and the variant axes are yours to change, and its bands come from your own verdicts: a task of your own never pools, and stats_pooled is always false for it. On a shared task (one of ours), a PATCH that changes any of them is refused with 422 shared_task_locked and changes nothing; re-sending a value equal to the stored one is not a change. To judge on your own rubric or prompts, create your own task, or send the change with allow_breaking_change: true, which makes it on a copy of your own where copies are available.

Reading it back

A slug that does not exist and a slug belonging to another account return the same 404.

What this costs

Defining a task costs nothing. Calling one is priced by the pricing terms, and the rule differs by charging mode.

pricing_mode: "byok" — the default. We charge one per-call fee for the request and nothing else, and only for a request that delivered; a request of yours that failed carries no fee from us (§1.4). Your provider is a separate matter, and it is not limited in that way: every model call we make to serve your request runs on your keys, so your provider bills you for the alternatives we try alongside the one we return, the evaluation calls that compare them, repairs of malformed output, and our retries after a provider error (§3.2). On a task of your own, exploration is a setting on the task — you can turn it off for a single request or for the task as a whole.

pricing_mode: "managed" on a task you define: billed at cost for as long as it exists. It is never put on a published rate (Pricing Terms §1.2), however much evidence it gathers. §1.2 applies as written: every model call in the chain appears on your invoice, per model — the alternatives, the evaluation calls that compare them, calls that failed, and our own retries after a provider error. While your task is still exploring those alternatives and evaluation calls are the larger share of that bill; switch exploration off, or narrow the task’s models, to hold it down.

A validator rejection retries the call on another model (see Validators); which pricing clause covers that retry is set out in the pricing terms, not here.

Limits

Control-plane writes (create, patch, register, deactivate) are rate-limited at 300 per hour, per account — every management key you hold shares the one budget — and return 429 with Retry-After: 3600 when the limit is reached. Reads are not counted against that limit.