Journal — Fronset Benchmark

Two models with almost the same price per token can cost very different amounts per task. In 55% of the model pairs we measured within 25% of each other on price, one cost at least twice as much per run as the other. The widest gap was 59×.

We priced September's 198,167 calls from Nova, our internal customer, three ways. Through Fronset: $766.79 with every fee. All on Claude Sonnet 5: $2,838.43. All on Claude Opus 5: $6,965.83. Judged quality was about the same.

The Fronset benchmark describes individual model-and-task cells in considerable detail and the grid around them hardly at all. Most model-and-task pairings hold no published result, and the results do not say why. Before you rely on a number, or read meaning into a missing one, check that the cell y

In the Fronset benchmark, naming topic clusters is a step where most models land close together; finding the themes in the first place is not. Every model with results on both steps scores lower on thematic topic discovery than on topic cluster labeling. And on discovery, the leading group includes

In the Fronset benchmark's model portfolio, every activity gets a default model and, only where the evidence supports one, a fallback. The small model can sit in either slot, and on several activities the fallback is left empty. What should move a call from the default to the fallback depends on how

On the Fronset benchmark's long-form writing work, the top of the table is a tie, not a leader. On reference-preserving analytical writing, the task closest to sustained, source-grounded drafting, three models sit inside overlapping score ranges, and so does a broad band below them. When the scores

Where the Fronset benchmark measured the same model on the same work in both batch and synchronous mode, batch cost less per task run in nearly every pair, and the quality score did not change. But the size of the saving swings widely by model and task, and in two pairs batch cost more. The batch pr

On the Fronset benchmark, the model with the highest quoted score on structured content summarization falls back on the neighbouring summarization and synthesis tasks, and models from lower in that ranking lead them instead. A ranking from one of these tasks does not carry over to another, even insi

On the Fronset benchmark's fact-extraction tasks, a lower failure rate does not on its own make a model the safer pick, because the way a model fails decides whether anyone notices. On refining factual claims, gemini-3.5-flash fails less often than MiniMax M3, but it mostly fails by dropping materia

On the Fronset benchmark's social-content selection tasks, average quality scores bunch so tightly that they often cannot tell models apart. Head-to-head results, which record how often a model's answer is judged the best of several answers to the same input, can separate models where the scores can

Claude Opus 5.5 scores 9.93 on structured output extraction, its highest quality figure in this material, yet on that task the score marks a tie at the ceiling rather than a lead. Its clearest win is elsewhere: matching profiles to a pool, where it was judged best on 18 of 26 occasions. Telling thos

For plumbing work, the usual reflex is to reach for the smallest member of a model family: the Lite, the Nano, the Haiku. On several utility tasks in the Fronset benchmark, that step down brings a clear drop in score, while on others it costs nothing measured. Which case applies depends on the task,

Since 2026-09-24, the Fronset benchmark records far more movement in how precisely scores are known than in the scores themselves. Confidence intervals closed across many tasks and confidence tiers shifted in both directions, while the means that did move mostly slipped, usually by small amounts. So

Models that score the relevance of a single item well do much worse when asked to score a set of content or a batch of short posts. Qwen 3.7 Plus tops the single-item list at 8.95 out of 10 but scores 5.69 on content sets, and no measured model reaches 7.5 on sets. If your pipeline scores many items

On engagement triage in the Fronset benchmark, Claude Opus 5.5 at $0.01 per task in sync mode is statistically tied with Claude Haiku 5.5 at $0.0003. At a bar that accepts any of the leading scores, that is the usual result: the leaders tie, and the defensible pick is the cheapest model in the tied

How long an LLM call takes in these findings depends first on the kind of work, and only after that on which model runs it. Qwen 3.8 Max answers language identification in 3.4 seconds but takes 246 seconds on topic sequence optimization. A latency decision therefore has to be made for each model-and

On some tasks in the Fronset Benchmark, the model you choose barely changes the result; on others, it changes it a lot. Language identification is the first kind: several leading models tie near a perfect score. Content domain suggestion is the second: scores are low and the best answers diverge wid

On many tasks in the Fronset benchmark, the leading models cannot be separated on quality, so price decides the choice. On engagement opportunity triage, Claude Haiku 5.5 scores 8.52 out of 10 (±0.33) at $0.0003 per task run in sync mode, while Claude Opus 5.5 scores 8.56 (±0.3) at $0.01. The except

For eleven models measured on the Fronset benchmark, the share of attempts that return usable output with no repair or retry ranges from 95.7% to 99.8%. On that measure the models are close to tied. What a pipeline needs to watch is what happens to the rest: for two models, some outputs needed a sec

Most latency figures in the Fronset benchmark are medians, and the median understates how long callers actually wait. Across 841,549 synchronous calls, the median is 5.5 seconds, but the 90th percentile is 51 seconds. A model with the lower median can also have the longer tail, so picking a model or

On the Fronset benchmark, a higher score is a real lead only when the two models' confidence intervals do not overlap. In the busy middle of the scale, between roughly 8 and 9, they usually do, and the higher score is then a tie rather than a lead. The numbers worth planning against are the ones tha

On the Fronset benchmark, a model's quality score is measured only on the calls that came back, so a high score can describe a small slice of the work you send it. GPT-6 Sol scores 8.98 out of 10 on publication title package generation but answers on only 10% of attempts. Read every score next to it

When judges (graders that score one output at a time against a published rubric) pick the best of several answers to the same inputs, length tilts the choice, but not in one consistent direction. In one set of 38 comparisons in the Fronset benchmark, the answer the judges picked was longer than the

No result in Fronset's financial-analysis category measures whether a trading decision was right. None measures returns or the accuracy of a financial conclusion either; the scores rate the answers models wrote on work next to that decision. For a developer building financial-analysis features, that

The published benchmark is broad enough to support decisions about **named model-task-mode combinations**. It is not a complete model leaderboard.

A production portfolio should be organized around transformations, not model reputations. The same model can be excellent after the direction is known and poor at deciding that direction in the first place.

This category contains an important divide. **Bounded classification**—identify a language, extract a known schema, analyze a document against a supplied profile—is often close to solved. **Interpretive judgment**—decide what is relevant, discover themes, choose sources, or match against a taxonomy—

Raising the quality bar does not reveal one universally best model. It changes which routes remain feasible—and how much the cheapest qualifying route costs.

DeepSeek V4 Flash has one of the broadest measured document-workflow profiles in this snapshot. The pattern is consistent: it is strongest when the direction and output structure are already defined. It is much less convincing when the system must discover the direction, make an open-ended judgment,

“Topic organization” sounds like one capability. In practice it contains at least six: discovering themes, suggesting a domain, generating a taxonomy, labeling clusters, assigning material to known sections, and sequencing the result. The model leader changes with the verb.

A benchmark score is trustworthy only for the exact task, operating mode, and population of calls it measures.

Theme discovery is a good example of why “cheapest model” is the wrong starting point. The task, serving mode, and quality bar must be fixed before the prices mean anything.

Batch can reduce the price of LLM work substantially. It does not make the wrong model right, preserve quality by definition, or satisfy a user who needs an immediate answer.

Benchmark scores become useful only after the workflow stage is named. Claude Sonnet 5 is strongly measured for language-led intake and metadata work, while Moonshot Kimi K3 is strongly measured for document structuring and synthesis. Neither profile establishes an end-to-end winner.

Latency is not a model-wide property. It belongs to a model–operation pair. In one Claude Sonnet 5 topic workflow, median latency ranges from 1.7 seconds for section assignment to 19 seconds for vetted-site selection. A single timeout for “topic analysis” is therefore a poor production design.

A model does not have one response time. An **operation** has a response time on a particular model—and the difference between operations can be larger than the difference between models.

A longer answer is not automatically better—and a shorter answer is not automatically more efficient. The useful question is whether the output budget matches the task.

Summarizing a document and reconciling competing perspectives are different jobs. The benchmark points to different leaders for each.

This snapshot does **not** redraw the model leaderboard. It changes two specific routes: prompt adaptation and language identification.

The benchmark does not support one default model for social content. General promotional messages are often effectively tied; the clearer advantages appear in specific deliverables such as voice definition, image prompts, newsletters, community selection, and engagement review.

A model can return clean output almost every time and still be the wrong model for the job. Meta Muse Spark 1.3 contributor produced a usable first response on 99.9% of 30,332 attempts, while Tencent Hy3 did so on 99.6% of 23,887 attempts. Those rates reveal how much repair and retry handling a work

There is no single “best model for finance” in the supplied evidence. The useful unit of comparison is the decision stage: gathering evidence, interpreting filings, refining claims, analyzing scenarios, and producing a recommendation.

For an unattended workflow, success is not “the API returned text.” Success is “the output can enter the next step without repair, retry, or human salvage.”

Structured extraction looks nearly solved if you read only the top scores. It looks much less settled once answer rate, repair burden, and failure type enter the decision.

Infrastructure workflows contain two very different kinds of AI work: routine transformations that many models perform similarly, and fact-sensitive judgments where failure type matters more than the headline score.

A high benchmark score can hide a weak production route. The score describes the answers a model returned; the answer rate tells you how often the workflow received one.

Long-form generation is a pipeline, not a single prompt. Planning, reference-preserving drafting, structured summarization, and final editorial copy produce different leaders.

Most teams pick one model and send it everything. Production data from our benchmark says that is the expensive option: on several kinds of work, the cheapest model on the shortlist scores within the margin of error of the priciest one, and the only thing that separates them is the bill. Here are tw

Most benchmark tables hand you one number per model and leave you to guess how much of it is signal. Ours attach a confidence interval to every score, and the reading rule is short: a model earns a rank only when its interval is tight enough, and a gap between two models only counts when their inter

If you build on LLM APIs, the most useful number in a benchmark is rarely the top score. It is the cheapest model that lands within a stated distance of that score. On financial catalyst and scenario analysis, the cheapest model that clears a 90%-of-best bar costs less than half of what the leader d

If you send summarization work to the biggest model you can afford, this benchmark suggests you are paying more for worse output. On turning structured content into summaries, GPT-5.4 Nano scored higher than GPT-5.5 at a lower price per run, and several other cheap models land close behind it. The s

If you route multi-perspective decision synthesis to the model with the highest score, you are paying roughly four times what an equally good option charges. The four top models on this task sit inside each other's confidence intervals, so there is no measurable quality gain for the extra spend. Pic

Most tasks in our benchmark end in a near-tie between the top two models, which is why "route on cost" is usually the right call. But a handful of jobs break that pattern: community selection, metadata paragraph improvement and content-domain suggestion produce real gaps between the leaders, with sp

If you use an LLM to discover themes, assign sections to topics or label clusters, the models on the Fronset Benchmark leaderboard land in the same mid-8s band and their confidence intervals overlap. That makes the score column mostly a tie and the cost column the real decision. The cost spread runs from 51

The task, not the model, decides how long an LLM call takes: the same model that answers in a few seconds on language detection needs minutes on topic matching. And within any one task, the slowest tenth of calls run several times longer than the median. A single global timeout sized for the typical

On relevance, classification and matching tasks, the model with the highest point score is usually not separated from the runner-up by more than its confidence interval. Once you treat overlapping intervals as a tie, the choices that actually move your bill are which cheaper model shares the tie, wh

If you choose an extraction model by its quality score alone, you will overestimate what actually ships. On the structured-data benchmark every score is measured only on the calls that came back, and some of the best-scoring models come back far less often than you would expect. Qwen 3.7 Plus scores

Running deferrable LLM work through a batch endpoint costs roughly half as much per task as running the same work synchronously, and on some tasks the saving is larger. Two things matter more than the mode switch, though: when models tie on quality, the cheapest option is sometimes a sync call on a

If you pick a model for analysis work by its judged score, the benchmark has an awkward message for you: the winner and the runner-up are usually a rounding error apart. On catalyst and scenario analysis, a judged win tells you far less than the leaderboard position suggests. The practical move is t

If you pick models by judged quality, the length of the winning answer is not a constant you can correct for. In our benchmark, judges picked the shorter answer on some tasks and the longer answer on others, and the direction flips even between closely related tasks. A single "penalize verbosity" or

Two findings from our long-form benchmarks stand out. The cheapest run that clears a production quality bar is MiniMax M3 writing newsletter copy, at a small fraction of a cent, and it lands only a quarter of a point below the best score in the whole set. The same cheap model then posts the weakest

A model that tops the benchmark on one everyday utility task can lose every head-to-head on the task next door. GPT-5.4 Nano and Tencent Hy3 both do exactly that in the current Fronset Benchmark data, and the averages that look safest also hide models that only answer some of the time. If you route utility

If you choose a model from a leaderboard, check how often it actually returns an answer before you trust its score. On turning source text into structured facts, gemini-3.5-flash scores 9.78 out of 10, but it only answers on 77% of attempts, and that score is measured only on the calls that came bac

If your pipeline has no human in the loop, the model that matters is the one that comes back with a usable answer on the first call, not the one with the best score on the calls it completed. On this cycle's evidence, the routes that pair solid quality with the highest answer rates are DeepSeek V4 F

If you are building topic clustering, labeling or taxonomy features on LLM APIs, the current benchmark evidence says there is no single model to pick. The same model that leads theme discovery collapses on content-domain suggestion, a model that clears the bar on taxonomy generation misses it on dis

A quality score on our benchmark only means something with three things attached: its confidence interval, the serving mode the cost was quoted in, and a flag saying whether that cost was measured or estimated. Read that way, the current data supports exactly one clean claim that paying more buys be

On short promotional posts, reply review and post selection, the leading models score so close that the benchmark's judges usually cannot separate them. That makes the model choice for most social and promotional work a question of latency and price, not quality. The one exception is picking which c

A benchmark score is one number measured under conditions your production system does not have. Two of those conditions matter more than the score itself. The score was measured only on the calls that came back, so it says nothing about the calls that did not. And it was measured without asking whet

Picking a model from a leaderboard bakes in two assumptions: that "better" means the same thing across your workload, and that speed is a property of the model. The latest Fronset Benchmark snapshot undercuts both. On the same benchmark, judges rewarded longer answers on one task and shorter answers on its

If you have ever set an LLM call's timeout by looking at a median latency and doubling it, this is for you. We run a public benchmark of production language tasks — extraction, triage, translation, summarization, long-form drafting — across a few dozen models, and the latency data keeps teaching the

Good enough is defined by the judged score on the specific measured task, in the execution mode—batch or sync—that the workflow actually uses, plus whether the model answers at all. A high score that requires salvage, retries, or manual repair is a poor production choice. This is one axis of compari

Since the previous cycle’s routing guidance for Qwen3.7 and Nemotron tiering, the current measurement introduces Claude Sonnet 5, minimax/minimax-m3, and Grok-4.5 across topic discovery, SEC filing interpretation, author-voice reproduction, and content drafting, with no reported score reversal or pr

There is no defensible universal winner across relevance scoring, classification, prioritization, and matching. The supplied results do not state how many tasks are in this category or how many models are measured on them, so those counts cannot be reported without inventing evidence. They do show a

For Infrastructure & Utility Work—work that names topic clusters, determines topic sequence, scores report relevance, analyzes filing chunks and full filings, writes analyst content tied to cited claims, refines claims, summarizes content, generates executive summaries, performs synthesis analysis,

For Financial Analysis & Trading Decisions, the practical quality bar is 8.0 out of 10. The evidence shows a trade-off rather than a universal winner: Grok-4.5 offers the strongest measured balance of trading-recommendation quality and synchronous cost, while GPT-5.5 and Gemini-3.5-flash are credibl

For this category, “good enough” is not a single headline score. A production model must deliver sufficiently strong output for the intended workflow—whether turning source material into analysis, summarizing content, generating sections, or filing chunks—while also meeting required standards for cl

The current evidence does not support a universal model ranking. It supports routing decisions by workflow stage, quality bar, serving mode, cost, and the probability that a request produces a usable answer. A conditional quality score is not enough when the model frequently fails to answer, returns

Production routing is a portfolio problem. The current evidence covers 54,627 scored judgments across 910 measured model-and-task combinations, covering 26 models on 52 tasks; the figures were published on a single date, and the measurement behind them is continuous. No single model wins enough of t

A model is good enough for production only when it clears the required quality bar and produces an accepted result reliably enough for the workflow. The measurements here show why those conditions must be evaluated together: a high score describes the quality of returned work, but not necessarily th

For production use, “good enough” is not simply a high score on answers that a model manages to return. The relevant bar is usable output on the first provider call: the response must arrive, conform to the workflow’s expected structure, and avoid salvage or a second repair call. The current evidenc

This snapshot’s broadest measured base for a single model belongs to Kimi-k3—not because it is the universal leader, but because the evidence covers it across demanding organization and interpretation stages with explicit score intervals, costs, and conditional measurements. I choose it as the deep-

For engineers selecting production models for topic organization and clustering, a practical quality bar is **8.0 out of 10**. The current excerpts cover naming clusters, open-ended discovery and theme generation, topic sequencing, section assignment, document-relevance scoring, client-topic matchin

The supplied results do not specify a universal quality threshold, nor do they state how many tasks or models make up the full Structured Data & Fact Extraction category. This comparison therefore uses a practical, task-level bar: a model must meet the extraction quality required by the workflow, wh

The supplied evidence does not state how many tasks are in the Social & Promotional Content category or how many models are measured across them. It also does not define a single category-wide quality threshold. The defensible production bar is therefore task-specific: choose the highest-quality opt

There is no defensible single winner across the long-form content measurements supplied here. The category-level task and model counts are not stated in the available evidence, so this comparison cannot provide them without inventing a figure. The results instead support a practical quality bar of *

Claude Sonnet 5 clears the 90% quality bar on 22 of 52 benchmark tasks, with real strength in financial analysis, extraction, and long-form writing. But it wins zero cheapest-good-enough slots, so the bench frames it as a premium quality pick, not a default value router.

Blended cost is the harness's final number — token cost, token shape, lane, and success rate folded into one honest price per useful result. This finale opens the box, shows the whole machine inside it, and closes the loop: the bench is the harness's readout, and its proof.

The list price of an LLM call is marketing. The invoice is truth. This is the harness's cost-measurement component — how it closes the loop between the rate card and the bill, and why the ★ "cheapest qualifier" means something because of it.

Multiplying tokens by the headline rate feels like the obvious way to compare model costs. It's wrong in three structural ways — and the harness's cost component exists to catch every one, so the price ranking the bench reads out is honest instead of the rate card's fiction.

Inside a mature harness, a model is an interchangeable part — and new parts arrive faster than organic traffic could ever score them. This is how the production harness places a brand-new model on the 0–10 scale quickly, by replaying real past work through it, and how deterministic sampling keeps the steady-state judging affordable.

Two models can post the same average score and still not be equally trustworthy. The confidence band is how the harness tells them apart — how it names what it doesn't yet know, gates which models earn live traffic, and why some cells on the bench show nothing at all.

The harness's evaluation layer is only as trustworthy as its judges — which are LLMs too. Here's the machinery, and where it stops: a panel from several labs, a model's own judge left out where the whole panel grades, strictness correction, drift detection and replayed judge swaps — turning "trust us" into "audit us," with the panel published, dated, on the bench.

"Quality" means two different things depending on the task, so the harness's evaluation component measures it two ways and combines them: deterministic verifiers catch what code can see, a judge panel grades what it can't, and a task-specific rubric turns the result into a single 0-10 score relative to the job the harness was built to do.

Two parts of the harness that keep it honest: latency-tolerant work runs at a steep discount in a batch lane, so the bench quotes both prices for the same interchangeable model — and every result is pinned to the exact prompt version that produced it, so scores never silently drift.

Every success rate on the bench is the readout of the harness's reliability layer — the machine around the model that catches malformed responses, classifies non-model failures, reroutes around broken providers, and refuses to count a model's misfires against it unfairly. Here's how that layer works.

Routing is the harness's model-selection component: every ★ on the heatmap is one runtime decision — pick the cheapest model that has earned the right to do this job. It works precisely because the harness holds prompts, validation, evaluation and cost fixed and swaps only the model, an interchangeable part. Here's the two-gate decision, and what catches it when it's wrong.

Leaderboards rank models. A harness is built around a job — its task definition — and ranks model-on-that-job. This is the first component of the machine, the fixed thing everything else attaches to, and why the same model can be your cheapest pick for one task and not even qualify for the next.

In Part 1 we argued the enterprise question isn't "which model is best?" but "which is good enough for the task?" This is the harness that answers it — the complete task-specific machine around the model, of which the bench is the visible readout — and the map for everything we dig into next.

Most enterprise AI teams ask which model is best. In production the better question is which model is good enough for a specific task — and the answer isn't a model at all, it's a harness: task-specific infrastructure that decomposes the work, holds the quality bar, and treats models as interchangeable hires. This origin post argues for task-level benchmarking as the honest readout of that harness.