Fronset Benchmark

Best LLMs for Content Domain Suggestion

Suggests one to three concise subject-domain labels for a content or analysis specification using its title, description, subject context, structure, and any existing domain vocabulary. Illustrative uses include classifying a software architecture proposal, consulting brief, supp

Cost mode:

Category: Relevance, Classification & Matching · Typical I/O: 5735→366 tokens

Runs on model group: All models. Inputs may reach any model in the group, not only the ranked models below.

Run this task on Fronset — request an invitation

Models

Frontier on this task: Thinking Machines Inkling Small at 7.59 / 10. Quality bar at 90%: 6.83.

point-estimate floor (CI low) · upper CI (less certain) · Bars sorted by blended cost; best-value model first. Greyed rows are MEDIUM+ models whose point estimate does not clear the bar.

Cost breakdown

ModelQualityConfidenceCost / 1k runsOverpayMode
Thinking Machines Inkling Small ★ best OpenRouter7.59 / 10 CI [7.14, 8.04]MEDIUM$4.17best valuebatch
Claude Opus 5.5 Anthropic7.20 / 10 CI [6.73, 7.68]MEDIUM$17.644.2xbatch

Overpay shows how much more you pay than the best-value model that clears the quality bar (marked ★) — the best-value good-enough option. "16x" means you overpay 16× — 16× that reference for no quality benefit above the bar. Typical call shape for this task: 5735 input tokens → 366 output tokens, EMA-tracked from production traffic. Cost is the observed, all-in $ per 1,000 task runs: each model's own measured usage on this task — output verbosity, thinking/reasoning tokens, cache reads and writes, and the spend on its billed failures — priced at current list rates and adjusted by the billing overhead we actually reconcile against provider invoices. Models that answer tersely cost what they actually cost; models that think at length pay for it. Not comparable to providers' advertised $/1M list rates — this is what running the task costs, not a per-token price.

Evaluation rubric

Judge semantic fit, coverage of the material’s principal subjects, label clarity, useful granularity, and reuse of equivalent existing labels. Penalize method/format labels, redundant synonyms, vague umbrella terms, and unsupported domains. Schema and item-count enforcement are out of scope.

Output schema

Every answer on this task is checked against this JSON Schema, whichever model wrote it. An answer that doesn't fit counts as a model failure, and the call is retried on another model.

{
  "$defs": {
    "NewDomainSpec": {
      "description": "One new Domain to create for AT content_domains.",
      "properties": {
        "description": {
          "anyOf": [
            {
              "type": "string"
            },
            {
              "type": "null"
            }
          ],
          "default": "",
          "title": "Description"
        },
        "name": {
          "title": "Name",
          "type": "string"
        }
      },
      "required": [
        "name"
      ],
      "title": "NewDomainSpec",
      "type": "object"
    }
  },
  "description": "LLM output for AT suggest_and_assign_content_domains.",
  "properties": {
    "domain_names_to_assign": {
      "items": {
        "type": "string"
      },
      "title": "Domain Names To Assign",
      "type": "array"
    },
    "new_domains_to_create": {
      "anyOf": [
        {
          "items": {
            "$ref": "#/$defs/NewDomainSpec"
          },
          "type": "array"
        },
        {
          "type": "null"
        }
      ],
      "default": null,
      "title": "New Domains To Create"
    }
  },
  "required": [
    "domain_names_to_assign"
  ],
  "title": "SuggestContentDomainsResult",
  "type": "object"
}

Prompt templates

The published template for this task.

default · LLMB_CONTENT_DOMAIN_SUGGESTION_SYSTEM + LLMB_CONTENT_DOMAIN_SUGGESTION_USER (0 calls in window)

System prompt

Identify the broad real-world domains the material is *about*, not the methods used to analyze it or the formats used to publish it. Prefer stable, recognizable labels; reuse a suitable supplied label before creating a near-duplicate. Return only domains supported by the input and do not infer a commercial vertical from weak wording. Treat empty optional values as absent and return only the requested result. Your response must conform exactly to this output schema: {schema_json_string}.

User prompt

Inputs — structure_summary: {structure_summary}; existing_domains: {existing_domains}; specification_name: {specification_name}; specification_description: {specification_description}; subject_name: {subject_name}; subject_code: {subject_code}. Use only these inputs to complete the task defined by the system prompt.