Fronset Benchmark

Models That Get It Right First Time

In the benchmark, MiniMax M3 failed outright on only 0.3% of its 23,451 attempts, yet 25.1% of those attempts needed repair before the answer could be used [^1][^2][^3][^4]. Count only outright failures and it looks highly available, while a quarter of its output still creates downstream work. Not failing, being usable first time and giving the best answer are three separate questions.

What a failure rate leaves out

A stricter measure than the failure rate is first-attempt usability: the share of a model’s attempts on which it returned a usable answer first time, with no repair and no retry. Several models post very high figures on it. Tencent Hy3 returned a usable first answer on 99.6% of its 23,887 attempts [^5][^6], and GPT-5.6 Luna on 98.8% of its 95,320 attempts [^4][^7][^8].

These are each model’s own record rather than a controlled comparison, and the attempts were not necessarily drawn from identical work. The differences among the top figures are small, so this rate alone should not decide between such models.

Where invalid output triggers a second provider call, that call adds cost and latency. For a model that often needs repair, the headline per-task figure therefore understates the end-to-end cost and elapsed time of a completed task. There are no repair cost or latency figures here for any model, so the size of that understatement is unknown.

Usable first time is not the same as best

First-attempt usability does not measure correctness directly, and it does not establish that a model produced the best answer for the task. The benchmark’s judged occasions speak to the second point: rival models answer the same live input, and one answer is chosen as the best.

GPT-5.6 Luna was chosen best in none of 42 judged occasions of writing a short promotional social post [^9][^10]. GPT-5.6 Terra was chosen best in none of 47 occasions of extracting claims from text [^11][^12]. Both models answer dependably, so reliability on the first attempt and suitability for quality-sensitive work can diverge. Why these models were not chosen is not established.

How a model fails matters as much as how often

On claim continuity classification, two models fail in different ways: Qwen 3.7 Plus mostly produces terse output, while Meta Muse Spark 1.3 mostly assigns the wrong class [^13]. A terse answer is usually visible and can be retried, whereas a wrong class can pass unnoticed. No counts are given for these failure modes, so how often each occurs is unknown.

What to do with this

  • Measure the share of output usable as returned, not just the outright failure rate, and validate parsing on structured paths.
  • If your workflow can’t absorb a retry, shortlist models with a high first-attempt rate, then test them on your own inputs against your output contract and quality bar.
  • Budget for the whole completion path, including repair calls and billed unusable tokens, and measure repair cost and latency directly.
  • Where a silently wrong answer would cause harm, treat it as more serious than a visible failure and put output checking ahead of any first-attempt figure.

The live benchmark at https://fronset.ai/benchmark/ shows each model’s task results, so you can set the first-pass quality bar your workflow actually needs.