Fronset Benchmark

Best Models for Content Summarization & Synthesis

On the benchmark’s structured content summarization task, the top models are statistically tied, and further down, MiniMax M3, at $0.0014 per task run, is tied on quality with Claude Opus 5.5 [^1][^2]. So neither leaderboard position nor price will pick your model for you. What the score doesn’t show is how a model fails: by inventing content, by being too brief, or by not answering at all [^2].

No model leads structured summarization

Scores are out of 10, and the ± figure after a score is its confidence interval: the margin of uncertainty around it. Where two models’ intervals overlap, the result is a tie, not a ranking.

GPT-5.6 Luna has the highest score with an interval, 8.91, ±0.12 after 181 independent judgments, but its interval overlaps those of GPT-5.6 Sol (8.9, ±0.23 after 40) and Kimi K3 (8.76, ±0.24 after 43), so the three are tied on this evidence [^1][^2]. Meta Muse Spark 1.3’s higher point score, 8.97, has no interval, so it can’t be placed against them with confidence [^2].

The cheapest model at a bar near 8.4

If your quality bar, the minimum score you need, is near 8.4, the cheapest model the benchmark supports is MiniMax M3, which scores 8.42, ±0.16 after 219 judgments [^1][^2]. Claude Opus 5.5 scores 8.14, ±0.32, at $0.02 per task run in batch mode (bulk processing, as opposed to sync calls answered on the spot) [^2].

Their intervals overlap, so the two are tied on this evidence, even though Claude Opus 5.5’s point score falls short of the bar [^2]. Against GPT-5.6 Luna, MiniMax M3 gives up roughly half a point. No per-task cost is given here for GPT-5.6 Luna on this task, so what that half point costs can’t be stated.

Below the top, models fail in different ways

Gemini 3.5 Flash mostly fails by inventing content, while MiniMax M3 mostly fails by being too brief [^2]. MiniMax M3’s judged failure rate, how often it was judged to have failed, is 22.1%, against 52.1% for Gemini 3.5 Flash [^2]. Beyond those figures, there is no basis here for sizing either failure mode.

A third way to fail is not answering at all. Among the options compared at a bar near 8.4, the lowest-cost one is Xiaomi MiMo V2.6 Flash, at $0.0008 per task in batch mode against $0.0016 in sync mode [^2]. It scores 7.98, ±0.3 after 31 judgments, but answers on only 23% of attempts, so it suits only a pipeline where a missed answer is tolerable [^2].

What to do with this

  • Treat a score with no ± interval as unranked, however high it is.
  • Set your quality bar first, then start from the cheapest model that clears it.
  • Before you switch, test that model on your own inputs for completeness as well as invented content.
  • Check how often a model answers at all, alongside its price and score.

Check the live task-level numbers at https://fronset.ai/benchmark/, where you can set the quality bar your workload requires and compare batch with sync pricing yourself.