Best LLMs for Public Response Generation

Category: Social & Promotional Content · Typical I/O: 2907→3210 tokens

Runs on model group: All models. Inputs may reach any model in the group, not only the ranked models below.

Run this task on Fronset — request an invitation

Models

Frontier on this task: Moonshot Kimi K3 at 8.45 / 10. Quality bar at 90%: 7.61.

point-estimate floor (CI low) · upper CI (less certain) · Bars sorted by blended cost; best-value model first. Greyed rows are MEDIUM+ models whose point estimate does not clear the bar.

Model	Quality score	CI low	Cost / 1k runs	vs best value
Xiaomi MiMo V2.6 Flash	7.65 / 10	7.21	$0.24	best value
Xiaomi MiMo V2.6 Pro	8.26 / 10	7.94	$0.90	3.7x more expensive
Tencent Hy3	7.87 / 10	7.59	$2.19	9.1x more expensive
Qwen 3.7 Plus	7.69 / 10	7.41	$4.95	21x more expensive
Claude Sonnet 5.5	7.69 / 10	7.31	$7.47	31x more expensive
Thinking Machines Inkling Small	8.07 / 10	7.76	$7.56	31x more expensive
Thinking Machines Inkling	7.86 / 10	7.56	$31.14	129x more expensive
Moonshot Kimi K3	8.45 / 10	8.28	$40.78	169x more expensive
Grok 4.7	7.64 / 10	7.25	$46.67	194x more expensive
GLM-5.3	7.74 / 10	7.39	$182.03	755x more expensive
GPT-6 Luna	7.21 / 10	6.75	$0.22	10% cheaper
Claude Haiku 5.5	7.43 / 10	6.96	$0.46	1.9x more expensive
GLM-5.3 Flash	7.51 / 10	7.02	$15.06	62x more expensive
Qwen 3.8 Flash	7.30 / 10	6.83	$9.78	41x more expensive
DeepSeek V4.1 Flash	7.60 / 10	7.21	$0.96	4x more expensive
Gemini 3.8 Flash	6.95 / 10	6.57	$3.42	14x more expensive
Tencent Hy4 Preview	7.25 / 10	6.77	$41.13	171x more expensive
Claude Opus 5.5	7.53 / 10	7.13	$15.48	64x more expensive
Qwen 3.8 Max	7.08 / 10	6.63	$67.97	282x more expensive
GPT-6 Sol	7.44 / 10	7.06	$3.61	15x more expensive
NVIDIA Nemotron-3 Nano 30B-A3B	5.91 / 10	5.44	$0.80	3.3x more expensive
Gemini 3.5 Flash Lite	6.46 / 10	6.03	$0.65	2.7x more expensive
NVIDIA Nemotron-3 Super 120B	6.39 / 10	5.96	$1.38	5.7x more expensive
MiniMax M3	7.43 / 10	7.06	$4.82	20x more expensive

Cost breakdown

Model	Quality	Confidence	Cost / 1k runs	Overpay	Mode
Xiaomi MiMo V2.6 Flash ★ Xiaomi MiMo	7.65 / 10 CI [7.21, 8.09]	MEDIUM	$0.24	best value	batch
Xiaomi MiMo V2.6 Pro Xiaomi MiMo	8.26 / 10 CI [7.94, 8.59]	MEDIUM	$0.90	3.7x	batch
Tencent Hy3 OpenRouter	7.87 / 10 CI [7.59, 8.16]	HIGH	$2.19	9.1x	batch
Qwen 3.7 Plus Alibaba Cloud (DashScope)	7.69 / 10 CI [7.41, 7.96]	HIGH	$4.95	21x	batch
Claude Sonnet 5.5 Anthropic	7.69 / 10 CI [7.31, 8.08]	MEDIUM	$7.47	31x	batch
Thinking Machines Inkling Small OpenRouter	8.07 / 10 CI [7.76, 8.37]	MEDIUM	$7.56	31x	batch
Thinking Machines Inkling OpenRouter	7.86 / 10 CI [7.56, 8.16]	MEDIUM	$31.14	129x	batch
Moonshot Kimi K3 best Moonshot AI	8.45 / 10 CI [8.28, 8.63]	RANKED	$40.78	169x	batch
Grok 4.7 xAI	7.64 / 10 CI [7.25, 8.02]	MEDIUM	$46.67	194x	batch
GLM-5.3 Z.AI	7.74 / 10 CI [7.39, 8.09]	MEDIUM	$182.03	755x	batch

Overpay shows how much more you pay than the best-value model that clears the quality bar (marked ★) — the best-value good-enough option. "16x" means you overpay 16× — 16× that reference for no quality benefit above the bar. Typical call shape for this task: 2907 input tokens → 3210 output tokens, EMA-tracked from production traffic. Cost is the observed, all-in $ per 1,000 task runs: each model's own measured usage on this task — output verbosity, thinking/reasoning tokens, cache reads and writes, and the spend on its billed failures — priced at current list rates and adjusted by the billing overhead we actually reconcile against provider invoices. Models that answer tersely cost what they actually cost; models that think at length pay for it. Not comparable to providers' advertised $/1M list rates — this is what running the task costs, not a per-token price.

Evaluation rubric

Judge direct relevance, evidence grounding, compliance with allowed and forbidden claims, conversational value, tone, platform fit, and appropriate abstention. Penalize generic compliments, self-promotion, and unsupported comparative claims.

Output schema

Every answer on this task is checked against this JSON Schema, whichever model wrote it. An answer that doesn't fit counts as a model failure, and the call is retried on another model.

{
  "description": "A drafted public reply for an engage-decision candidate.",
  "properties": {
    "reply_text": {
      "description": "The reply, in the persona voice. Put the literal token \u003cLINK\u003e where a tracked Fronset benchmark link belongs (only if genuinely useful).",
      "title": "Reply Text",
      "type": "string"
    },
    "uses_link": {
      "default": false,
      "description": "True if the reply includes the \u003cLINK\u003e token.",
      "title": "Uses Link",
      "type": "boolean"
    }
  },
  "required": [
    "reply_text"
  ],
  "title": "EngagementReplyDraft",
  "type": "object"
}

Prompt templates

The published template for this task.

default · LLMB_PUBLIC_RESPONSE_GENERATION_SYSTEM + LLMB_PUBLIC_RESPONSE_GENERATION_USER (184 calls in window)

System prompt

Respond directly to the supplied message and add genuine conversational value. Use only allowed evidence, respect forbidden claims, and apply channel_profile, policy_profile, and voice_profile as the complete source of length, tone, formatting, safety, and release behavior. If the evidence cannot support the requested angle, return the schema-defined abstention. Treat empty optional values as absent and return only the requested result. Your response must conform exactly to this output schema: {schema_json_string}.

User prompt

Inputs — operator_instructions: {operator_instructions}; platform: {platform}; max_length: {max_length}; author_handle: {author_handle}; post_text: {post_text}; angle: {angle}; approved_claims: {approved_claims}; forbidden_claims: {forbidden_claims}; covered_models: {covered_models}; journal_articles: {journal_articles}; model_ranking: {model_ranking}; bench_claims: {bench_claims}; channel_profile: {channel_profile}; policy_profile: {policy_profile}; voice_profile: {voice_profile}. Use only these inputs to complete the task defined by the system prompt.