Before Blaming the Model, Check Whether the Whole Task Moved
Before Blaming the Model, Check Whether the Whole Task Moved
Since 2026-10-16, scores on the benchmark have drifted by fractions of a point, and more moved down than up. Most of that drift lines up by task, not by model. On several tasks, every model measured moved the same way. A model that slips on every task it runs is a different pattern, and that is the one worth re-testing.
The difference matters because a lower score is easy to read as a regression. If you route on benchmark results, or track them for a model already in production, you need to know whether the model changed or every model’s score on that task did.
When a task moves as a block
Five tasks show movement shared across models.
| Task | Movement across models | Examples |
|---|---|---|
| Catalyst and scenario analysis | Down an identical 0.07 for several models | Claude Opus 5.5 to 8.14, Grok 4.7 to 7.97 [^1] |
| Extracting structure from documents | Down for every model measured | Claude Haiku 5.5 down 0.12 to 8.35, Claude Opus 5.5 down 0.10 to 7.85 [^2] |
| Identifying a text’s language | Down for every model with a reported change | Declines of 0.07 to 0.28 [^3] |
| Writing public responses | Up for most models measured | DeepSeek V4.1 Flash up 0.24 to 7.85, Claude Sonnet 5.5 up 0.22 to 7.92 [^4] |
| Analysing profiled documents | Up | Grok 4.7 up 0.26 to 9.17, GPT-6 Sol up 0.25 to 8.58 [^5] |
When the same move appears across many models, the simplest reading is a shift tied to the task and not to any one model. The identical size on catalyst and scenario analysis is the strongest hint, because separate changes in separate models would be unlikely to land on the same figure.
That reading is an interpretation. The findings report what moved, not why it moved.
A shift at the task level also leaves the order between models alone. The material reports no reversal in standing between two models. So on these tasks the model you picked before is still the one you would pick, unless a shared drop has taken it below the quality bar you need.
The model that slipped everywhere
Xiaomi MiMo V2.6 Flash is the clearest example. It slipped on every task measured, including classifying claim continuity, where it fell 0.13 to 8.14 [^6].
Where a task moved as a block, it moved further than the block. On catalyst and scenario analysis it fell 0.13 to 6.92, while several other models fell by the shared 0.07 [^1]. On language identification, its drop of 0.28 to 9.86 is the largest reported [^3].
That combination separates a candidate regression from a task-level shift. The model is down on every task, and by more than the shared movement wherever one exists. It is still only a candidate. The moves are tenths of a point, the findings give no cause, and 9.86 still sits near the top of the scale for language identification [^3].
A drift check you can run
Compare across models first. Put the moved score next to other models’ scores on the same task. If they moved the same way by a similar amount, treat it as a task-level shift and keep your routing, as long as the new level still clears your bar.
Then look across tasks. A model that fell on every task it runs is the one to re-test, even when each drop is small.
Subtract the shared move. Where a task moved as a block, only the part of a model’s change beyond the shared movement says anything about that model.
Re-test on your own inputs before switching. Drift of this size is a reason to re-check, not evidence that another model is now better.
The caveat that matters is that this is a reading of patterns, not a diagnosis. With no stated cause behind any of the movements, a shared move is a reason to leave routing alone and a model-wide slip is a reason to measure again. Neither is a verdict.
Current figures for every task and model named here are at https://fronset.ai/benchmark/. Before deciding a model has regressed, open the task its score belongs to and check whether the other models on it moved too.