How MiniMax M3 and gemini-3.5-flash Fail Differently on Summaries
MiniMax M3 fails less often than gemini-3.5-flash when turning content into structured summaries: a judged failure rate of 23.0% against 52.3%, on the same task under the same rubric [^1]. The more useful difference is how each model fails, because that decides whether a bad output costs you a retry or slips through as a plausible wrong answer.
Two ways to get a summary wrong
When gemini-3.5-flash fails at summarization, it mostly invents content. When MiniMax M3 fails, it is mostly too brief [^1]. The rubric counts both as failures, but they behave very differently once the output leaves the model.
A summary that is too short is usually obvious. A length check or a quick look flags it, and you send the request again. A summary with made-up content can look complete and well formed, so nothing in your pipeline necessarily notices it.
That split into visible and silent failures is our interpretation, not part of the measured results. The benchmark names each model’s dominant failure mode; it does not label which modes are cheap or costly. The reasoning is simple, though: a failure you can see costs a retry, and a failure you can’t see costs whatever happens downstream.
The pattern holds on other tasks
The same two models diverge in a similar way on two other kinds of work:
| Work | MiniMax M3 | gemini-3.5-flash |
|---|---|---|
| Structured summarization | Mostly too brief [^1] | Mostly invents content [^1] |
| Scoring how relevant a document is | Mostly marks everything low [^2] | Mostly answers something else entirely [^2] |
| Refining factual claims | Mostly “other” [^3] | Mostly drops material [^3] |
Across all three, gemini-3.5-flash’s dominant modes are the silent kind: invention, answering a different question, dropping material [^1][^2][^3]. MiniMax M3’s all-low relevance scores are, like its short summaries, the kind a check can catch [^1][^2]. Its failures on claim refinement are labelled only “other” [^3], which says nothing about whether they would be visible.
What to do with it
For summarizing content into structured facts, where a wrong but plausible output would go unnoticed downstream, MiniMax M3 is the defensible choice on these figures. It fails less than half as often, and its typical failure is one you can detect and resend [^1].
The same logic favors MiniMax M3 for relevance scoring, because a batch of all-low scores is easier to catch than an answer to a different question [^2]. Claim refinement is less clear: gemini-3.5-flash’s dropped material is silent, while MiniMax M3’s “other” is unspecified [^3].
Whichever model you use, it pays to know its dominant failure mode and to check for that one specifically. A minimum-length check costs almost nothing and turns brevity into a retry. If your pipeline already retries short output, the gap that remains between these two models is mostly about silent error, and strong retry-and-check tooling narrows it further.
What these results don’t settle
Only the summarization comparison carries a failure rate. The relevance scoring and claim refinement results report each model’s dominant failure mode but no rates [^2][^3]. They show how these models break on those tasks, not how often, so the table supports a judgment about failure type there, not about which model fails more.
The live benchmark is at https://fronset.ai/benchmark/, where you can set the quality bar you need and compare models on the tasks you run, since the figures here are one reading of a bar you may want to move. The method is documented at https://fronset.ai/benchmark/methodology/.