Best LLMs for Public Response Generation
Drafts a concise public response to an input message using a supplied angle, evidence boundaries, channel profile, policy profile, and operator instructions. Illustrative uses include drafting a response to a customer complaint, developer issue, analyst question, partner inquiry,
Run this task on Fronset — request an invitation
Models
Frontier on this task: Xiaomi MiMo V2.6 Pro at 8.48 / 10. Quality bar at 90%: 7.63.
point-estimate floor (CI low) · upper CI (less certain) · Bars sorted by blended cost; best-value model first. Greyed rows are MEDIUM+ models whose point estimate does not clear the bar.
| Model | Quality score | CI low | Cost / 1k runs | vs best value |
|---|---|---|---|---|
| Xiaomi MiMo V2.6 Flash | 7.87 / 10 | 7.44 | $0.24 | best value |
| Claude Haiku 5.5 | 7.64 / 10 | 7.17 | $0.45 | 1.9x more expensive |
| Xiaomi MiMo V2.6 Pro | 8.48 / 10 | 8.16 | $0.93 | 3.9x more expensive |
| DeepSeek V4.1 Flash | 7.85 / 10 | 7.46 | $0.96 | 4.1x more expensive |
| Tencent Hy3 | 7.84 / 10 | 7.56 | $2.12 | 9x more expensive |
| GPT-6 Sol | 7.65 / 10 | 7.25 | $3.91 | 17x more expensive |
| Thinking Machines Inkling Small | 8.06 / 10 | 7.76 | $7.44 | 31x more expensive |
| Claude Sonnet 5.5 | 7.92 / 10 | 7.54 | $7.47 | 32x more expensive |
| Claude Opus 5.5 | 7.77 / 10 | 7.38 | $15.41 | 65x more expensive |
| Thinking Machines Inkling | 7.86 / 10 | 7.55 | $27.04 | 114x more expensive |
| Moonshot Kimi K3 | 8.41 / 10 | 8.24 | $45.28 | 192x more expensive |
| Grok 4.7 | 7.85 / 10 | 7.46 | $49.28 | 208x more expensive |
| GLM-5.3 | 7.83 / 10 | 7.51 | $178.15 | 754x more expensive |
| GPT-6 Luna | 7.42 / 10 | 6.95 | $0.37 | 1.6x more expensive |
| GLM-5.3 Flash | 7.58 / 10 | 7.09 | $14.95 | 63x more expensive |
| Qwen 3.8 Flash | 7.30 / 10 | 6.83 | $4.91 | 21x more expensive |
| Gemini 3.8 Flash | 6.99 / 10 | 6.61 | $3.57 | 15x more expensive |
| Tencent Hy4 Preview | 7.35 / 10 | 6.90 | $41.82 | 177x more expensive |
| Qwen 3.8 Max | 7.24 / 10 | 6.77 | $65.52 | 277x more expensive |
| NVIDIA Nemotron-3 Nano 30B-A3B | 5.90 / 10 | 5.44 | $0.93 | 3.9x more expensive |
| Gemini 3.5 Flash Lite | 6.46 / 10 | 6.03 | $0.65 | 2.7x more expensive |
| NVIDIA Nemotron-3 Super 120B | 6.37 / 10 | 5.95 | $1.37 | 5.8x more expensive |
| Qwen 3.7 Plus | 7.63 / 10 | 7.35 | $8.26 | 35x more expensive |
| MiniMax M3 | 7.39 / 10 | 7.02 | $4.84 | 20x more expensive |
Cost breakdown
| Model | Quality | Confidence | Cost / 1k runs | Overpay | Mode |
|---|---|---|---|---|---|
| Xiaomi MiMo V2.6 Flash ★ Xiaomi MiMo | 7.87 / 10 CI [7.44, 8.29] | MEDIUM | $0.24 | best value | batch |
| Claude Haiku 5.5 Anthropic | 7.64 / 10 CI [7.17, 8.11] | MEDIUM | $0.45 | 1.9x | batch |
| Xiaomi MiMo V2.6 Pro best Xiaomi MiMo | 8.48 / 10 CI [8.16, 8.80] | MEDIUM | $0.93 | 3.9x | batch |
| DeepSeek V4.1 Flash DeepSeek | 7.85 / 10 CI [7.46, 8.23] | MEDIUM | $0.96 | 4.1x | batch |
| Tencent Hy3 OpenRouter | 7.84 / 10 CI [7.56, 8.13] | HIGH | $2.12 | 9x | batch |
| GPT-6 Sol OpenAI | 7.65 / 10 CI [7.25, 8.04] | MEDIUM | $3.91 | 17x | batch |
| Thinking Machines Inkling Small OpenRouter | 8.06 / 10 CI [7.76, 8.37] | MEDIUM | $7.44 | 31x | batch |
| Claude Sonnet 5.5 Anthropic | 7.92 / 10 CI [7.54, 8.29] | MEDIUM | $7.47 | 32x | batch |
| Claude Opus 5.5 Anthropic | 7.77 / 10 CI [7.38, 8.16] | MEDIUM | $15.41 | 65x | batch |
| Thinking Machines Inkling OpenRouter | 7.86 / 10 CI [7.55, 8.16] | MEDIUM | $27.04 | 114x | batch |
| Moonshot Kimi K3 Moonshot AI | 8.41 / 10 CI [8.24, 8.59] | RANKED | $45.28 | 192x | batch |
| Grok 4.7 xAI | 7.85 / 10 CI [7.46, 8.24] | MEDIUM | $49.28 | 208x | batch |
| GLM-5.3 Z.AI | 7.83 / 10 CI [7.51, 8.16] | MEDIUM | $178.15 | 754x | batch |
Overpay shows how much more you pay than the best-value model that clears the quality bar (marked ★) — the best-value good-enough option. "16x" means you overpay 16× — 16× that reference for no quality benefit above the bar. Typical call shape for this task: 2930 input tokens → 3242 output tokens, EMA-tracked from production traffic. Cost is the observed, all-in $ per 1,000 task runs: each model's own measured usage on this task — output verbosity, thinking/reasoning tokens, cache reads and writes, and the spend on its billed failures — priced at current list rates and adjusted by the billing overhead we actually reconcile against provider invoices. Models that answer tersely cost what they actually cost; models that think at length pay for it. Not comparable to providers' advertised $/1M list rates — this is what running the task costs, not a per-token price.
Evaluation rubric
Judge direct relevance, evidence grounding, compliance with allowed and forbidden claims, conversational value, tone, platform fit, and appropriate abstention. Penalize generic compliments, self-promotion, and unsupported comparative claims.
Output schema
Every answer on this task is checked against this JSON Schema, whichever model wrote it. An answer that doesn't fit counts as a model failure, and the call is retried on another model.
{
"description": "A drafted public reply for an engage-decision candidate.",
"properties": {
"reply_text": {
"description": "The reply, in the persona voice. Put the literal token \u003cLINK\u003e where a tracked Fronset benchmark link belongs (only if genuinely useful).",
"title": "Reply Text",
"type": "string"
},
"uses_link": {
"default": false,
"description": "True if the reply includes the \u003cLINK\u003e token.",
"title": "Uses Link",
"type": "boolean"
}
},
"required": [
"reply_text"
],
"title": "EngagementReplyDraft",
"type": "object"
}Prompt templates
The published template for this task.
default ·
LLMB_PUBLIC_RESPONSE_GENERATION_SYSTEM +
LLMB_PUBLIC_RESPONSE_GENERATION_USER
(220 calls in window)
System prompt
Respond directly to the supplied message and add genuine conversational value. Use only allowed evidence, respect forbidden claims, and apply channel_profile, policy_profile, and voice_profile as the complete source of length, tone, formatting, safety, and release behavior. If the evidence cannot support the requested angle, return the schema-defined abstention. Treat empty optional values as absent and return only the requested result. Your response must conform exactly to this output schema: {schema_json_string}.
User prompt
Inputs — operator_instructions: {operator_instructions}; platform: {platform}; max_length: {max_length}; author_handle: {author_handle}; post_text: {post_text}; angle: {angle}; approved_claims: {approved_claims}; forbidden_claims: {forbidden_claims}; covered_models: {covered_models}; journal_articles: {journal_articles}; model_ranking: {model_ranking}; bench_claims: {bench_claims}; channel_profile: {channel_profile}; policy_profile: {policy_profile}; voice_profile: {voice_profile}. Use only these inputs to complete the task defined by the system prompt.