A Near-Zero LLM Failure Rate Can Still Hide a Heavy Repair Load
A Near-Zero LLM Failure Rate Can Still Hide a Heavy Repair Load
A model that almost never fails can still hand back output you cannot use. On the Fronset benchmark, the MiniMax record with the lowest failure rate also has the smallest share of output usable as returned [^1][^2].
A failure rate counts one outcome out of three
In these records, every attempt ends in one of three ways. It fails outright, it returns output that needs repair, or it returns output that can be used as it arrives, with no repair call and no retry. A failure rate counts only the first, so output that needs repair never shows up in it.
The middle outcome is not free. Each repair is a second call, so it adds cost and latency that a headline per-task price does not show. The benchmark reports neither the cost nor the latency of repair, so a model can look cheap per task and still be expensive in practice. For a pipeline, the number that matters is the share usable as returned.
Ranked by failure rate, the order is backwards
The benchmark splits attempts into all three outcomes for MiniMax M2.5 and, in two separate records, for MiniMax M3:
| Record | Attempts | Fails outright | Needs repair | Usable as returned |
|---|---|---|---|---|
| MiniMax M2.5 [^1] | 40,876 | 0.4% | 34.3% | 65.4% |
| MiniMax M3, record A [^1] | 35,961 | 5.0% | 18.1% | 76.9% |
| MiniMax M3, record B [^2] | 25,043 | 0.9% | 25.8% | 73.3% |
By failure rate, MiniMax M2.5 looks like the most dependable of the three. By usable output, it is the least. Across these records the two orderings are exact opposites: the fewer the outright failures, the larger the repair share and the smaller the usable share.
Three records are too few to call that a rule, and the benchmark does not explain it. They are enough to show that a low failure rate is not evidence of usable output.
The gap to other models is wide. The six models with first-time usability figures range from 98.1% for Claude Sonnet 5.5, over 4,214 attempts [^3], down to 95.7% for Qwen 3.6 Plus, over 3,485 [^1]. The largest sample, Gemini 3.1 Flash Lite, reaches 97.1% across 202,859 attempts [^1].
What to measure instead
This section is our interpretation of the results, not a measured outcome.
- Log all three outcomes for every call. Set reliability targets on the usable share, not the error rate.
- Without a repair step, treat repair-needed output as failed. For MiniMax M2.5, plan around its usable share, not its failure rate.
- With a repair step, budget a second call per repaired attempt. Measure its cost and latency on your own traffic.
- Check against your own output contract. The first-time figures are not a schema-by-schema check.
The caveat: the records disagree
The two MiniMax M3 records disagree on every column of the table, and the material does not say what work each one covers. Neither is the MiniMax M3 figure, and averaging them into one reliability number would hide that disagreement.
The first-time figures for the six other models are not tied to kinds of work either. The gap between them and the MiniMax records is therefore not a like-for-like comparison. What holds in every record is narrower: its failure rate alone would not have told you how much of its output was usable.
The live benchmark at https://fronset.ai/benchmark/ lets you set the quality bar your workflow needs and compare models against it. When a candidate’s failure rate looks near zero, check how much of its output arrives usable before you route production traffic to it.