Fronset Benchmark

Best LLMs for Community Content Promotion Generation

Decides whether a content item is appropriate for each supplied community and, for suitable communities, drafts a substantive contribution using caller-supplied channel and community rules. Illustrative uses include preparing appropriate contributions for an open-source community

Cost mode:

Category: Social & Promotional Content · Typical I/O: 7274→1880 tokens

Runs on model group: All models. Inputs may reach any model in the group, not only the ranked models below.

Run this task on Fronset — request an invitation

Models

Frontier on this task: Claude Opus 5.5 at 8.58 / 10. Quality bar at 90%: 7.72.

point-estimate floor (CI low) · upper CI (less certain) · Bars sorted by blended cost; best-value model first. Greyed rows are MEDIUM+ models whose point estimate does not clear the bar.

Cost breakdown

ModelQualityConfidenceCost / 1k runsOverpayMode
Qwen 3.7 Plus ★ Alibaba Cloud (DashScope)7.75 / 10 CI [7.51, 7.99]HIGH$7.36best valuebatch
GLM-5.3 Flash Z.AI8.00 / 10 CI [7.62, 8.38]MEDIUM$14.462xbatch
Claude Sonnet 5.5 Anthropic7.82 / 10 CI [7.34, 8.29]MEDIUM$20.602.8xbatch
Claude Opus 5.5 best Anthropic8.58 / 10 CI [8.24, 8.92]MEDIUM$46.206.3xbatch
Grok 4.7 xAI8.16 / 10 CI [7.70, 8.62]MEDIUM$59.588.1xbatch
Moonshot Kimi K3 Moonshot AI8.03 / 10 CI [7.75, 8.31]HIGH$97.5113xbatch
GLM-5.3 Z.AI8.24 / 10 CI [7.86, 8.61]MEDIUM$188.5226xbatch

Overpay shows how much more you pay than the best-value model that clears the quality bar (marked ★) — the best-value good-enough option. "16x" means you overpay 16× — 16× that reference for no quality benefit above the bar. Typical call shape for this task: 7274 input tokens → 1880 output tokens, EMA-tracked from production traffic. Cost is the observed, all-in $ per 1,000 task runs: each model's own measured usage on this task — output verbosity, thinking/reasoning tokens, cache reads and writes, and the spend on its billed failures — priced at current list rates and adjusted by the billing overhead we actually reconcile against provider invoices. Models that answer tersely cost what they actually cost; models that think at length pay for it. Not comparable to providers' advertised $/1M list rates — this is what running the task costs, not a per-token price.

Evaluation rubric

Judge community relevance, rule compliance, source grounding, usefulness independent of promotion, tone fit, and sound skip decisions. Penalize disguised advertising, invented claims, generic cross-post copy, and attempts to evade explicit community restrictions.

Output schema

Every answer on this task is checked against this JSON Schema, whichever model wrote it. An answer that doesn't fit counts as a model failure, and the call is retried on another model.

{
  "$defs": {
    "SubredditPost": {
      "properties": {
        "body": {
          "default": "",
          "description": "Full Reddit post body in markdown, tailored to this subreddit. MUST include \u003cghost_url\u003e placeholder for the report link. Required if should_post=True.",
          "title": "Body",
          "type": "string"
        },
        "should_post": {
          "description": "Whether this report is relevant enough for this subreddit",
          "title": "Should Post",
          "type": "boolean"
        },
        "skip_reason": {
          "default": "",
          "description": "Brief reason if should_post=False",
          "title": "Skip Reason",
          "type": "string"
        },
        "subreddit": {
          "description": "Subreddit name (without r/ prefix)",
          "title": "Subreddit",
          "type": "string"
        },
        "title": {
          "default": "",
          "description": "Subreddit-specific title, max 300 chars. Required if should_post=True.",
          "title": "Title",
          "type": "string"
        }
      },
      "required": [
        "subreddit",
        "should_post"
      ],
      "title": "SubredditPost",
      "type": "object"
    }
  },
  "properties": {
    "subreddit_posts": {
      "description": "One entry per target subreddit",
      "items": {
        "$ref": "#/$defs/SubredditPost"
      },
      "title": "Subreddit Posts",
      "type": "array"
    }
  },
  "required": [
    "subreddit_posts"
  ],
  "title": "AutoRedditPostsOutput",
  "type": "object"
}

Prompt templates

The published template for this task.

default · LLMB_COMMUNITY_CONTENT_PROMOTION_GENERATION_SYSTEM + LLMB_COMMUNITY_CONTENT_PROMOTION_GENERATION_USER (0 calls in window)

System prompt

Evaluate each target community independently. Create a contribution only when the content is relevant and permitted by the supplied community rules. Draft a useful community-native contribution grounded in source_material; apply channel_profile for platform behavior, titles, labels or tags, links, disclosure, formatting, automation, and promotional constraints, and apply optional voice_profile for voice. Prefer the configured skip result to fabricating relevance or bypassing a rule. Treat empty optional values as absent and return only the requested result. Your response must conform exactly to this output schema: {schema_json_string}.

User prompt

Inputs — content_item: {content_item}; source_material: {source_material}; target_communities: {target_communities}; channel_profile: {channel_profile}; voice_profile: {voice_profile}. Use only these inputs to complete the task defined by the system prompt.