Fronset Benchmark

How Long Will This Take?

Medians hide slow calls: a latency budget built on the typical call fails on the slowest ones, and a model that looks quicker at the median can have the longer tail. Batch jobs show the same pattern on a longer clock [^1][^2].

The median hides a long tail

Across 699,136 synchronous calls in the benchmark, the kind where the caller waits for each answer, the median took 5.0 seconds but one call in ten took 51 seconds or longer [^1][^2][^3][^4][^5][^6][^7][^8][^9][^10][^11][^12][^13][^14][^15][^16][^17][^18][^19][^20][^21][^22][^23][^24][^25].

The 90th percentile is the time within which nine calls in ten finish. On claim continuity classification, Tencent Hy3 took 40 seconds at the median and 171 seconds at the 90th percentile, over 194 calls [^26]. An interactive path on that model should be budgeted against the 171 seconds, not the 40 [^26].

A quicker median can come with a longer tail

On the same task, NVIDIA Nemotron 3.5 Lightning took 18 seconds at the median over 212 calls, against 27 seconds over 173 calls for Thinking Machines Inkling Small [^26]. At the 90th percentile the order flipped: 173 seconds for Nemotron 3.5 Lightning and 85 seconds for Inkling Small [^26].

On this task, neither model is faster outright. Nemotron 3.5 Lightning is the choice when the median is the constraint, and Inkling Small when the slow tail is.

Past the tail: calls that miss the deadline

Deadline misses are a separate source of delay from slow answers. Against a 600-second per-call deadline, gpt-5.4-mini missed it on 14.8% of its 2,413 attempts [^1][^2], and Qwen 3.8 Max on 13.5% of its 2,746 attempts [^27].

No tokens were metered for those misses, so they carry no token cost and show up as lost answers and waiting time rather than as spend [^1][^2][^27]. How long to wait is a setting of the benchmark, not a property of either model.

Batch usually returns in minutes, but plan for the hour

Batch is the slow lane, for work that can wait, and providers publish a window of up to 24 hours [^1][^2]. In the benchmark, half of batches came back within 5.5 minutes and 75.1% within 15 minutes [^1][^2].

The planning risk is in the tail. Under a one-hour deadline, 4.1% of everything submitted was cancelled and re-sent, and 5.6% of the batches that completed took longer than an hour [^1][^2].

Batch reliability is a property of the provider rather than the model [^1][^2]. Under the same deadline, cancel-and-resend rates were 6.4% for anthropic on 4,231 submitted batches, 4.7% for xiaomi on 3,840 and 3.9% for gemini on 24,335 [^1][^2]. These rates come without confidence intervals, or uncertainty ranges, so the gaps are measured differences rather than a settled ranking.

What to do with this

  • Before committing to a model for an interactive path, measure its 90th percentile on your own task; few of the results here include one.
  • Track deadline misses separately from spend, and check timeout and retry handling before relying on a model that misses often.
  • Use batch for work that can wait an hour or more and absorb a re-send; where a few minutes matter, plan on synchronous calls.

These figures measure time only, not answer quality. See the live benchmark at https://fronset.ai/benchmark/ for quality results and for batch and synchronous pricing.