Two Things the Median Latency Won’t Tell You

If you have ever set an LLM call’s timeout by looking at a median latency and doubling it, this is for you. We run a public benchmark of production language tasks — extraction, triage, translation, summarization, long-form drafting — across a few dozen models, and the latency data keeps teaching the same two lessons. First, the median is not the number that sets your timeout; the tail is, and on these tasks the tail runs three to seven times the median. Second, the largest latency differences are not between model families but between variants inside a family, on work where judged quality is effectively tied. Both are decisions you get to make, and neither shows up on a model card.

Start with the deadline, because it is yours

Every synchronous call in our benchmark gets 600 seconds to answer. That number is our setting, not a property of any model, and it matters because it defines what “missed” means. Against that deadline, GPT-5.6 Sol misses on 25.6% of its 4,061 attempts [^1][^2][^3] and gpt-5.4-mini on 14.8% of its 2,413 [^4][^5]. Our metering records zero tokens for every one of those misses, because the SDK dropped the request before any usage report came back. That is not the same as the call being free. The timeout lives on the client; the provider may well have finished generating the answer we walked away from, and may invoice for it. How much of an abandoned request gets billed depends on the provider and on whether the call was streaming, but the safe assumption is that some of it is. So a miss costs the full wait, a retry on top of it, and potentially the tokens of an answer nobody collected — a cost that never appears in your own token accounting and shows up only as a gap between metered usage and the invoice.

The causes are instructive because no single one dominates. Timeouts account for 18.0% of misses, measured against the 600-second deadline we set, rate limiting for 19.1% and parse failures for 12.0% [^4][^5]. Only the parse failures are unambiguously the model’s fault. The timeout is a number we chose, and a rate limit is a quota applied to our traffic rather than a property of the model, which is another way of saying that a good share of what looks like model unreliability is a deployment choice.

One piece of housekeeping is worth copying into your own dashboards. In our data, 0.014% of successful synchronous calls (101 of them) ran past the 600-second deadline, and we exclude them from every latency figure, because they are stalls rather than slow answers, and a single stall moves a 90th percentile further than a hundred honest slow calls [^6][^4][^7][^8][^9][^10][^11][^12][^13][^14][^15][^16][^17][^18][^19][^20][^21][^22][^23][^24][^25][^26][^27][^28][^5]. If your observability stack folds hung connections into p90, your p90 describes the hangs, not the answers.

The tail is three to seven times the median

Here is the shape that repeats across every family we measure. GPT-5.4 Nano summarizing content answers in 12 seconds at the median but 38 seconds at the 90th percentile, over 1,390 calls [^6][^20][^29]. Size an interactive path on the median and one call in ten blows through it by that margin. It is not a quirk of one small model, and it is not confined to slow tasks:

Sources for the table: topic-report relevance [^6][^13][^30], with DeepSeek V4 Flash additionally [^4][^31]; content summarization [^6][^20][^29]; translation [^6][^26][^32]; claim generation [^6][^4][^33]; language detection [^6][^34]. All calls were made under the same 600-second deadline, with stalls excluded as described above.

The practical consequence is blunt. A timeout set at twice the median, the most common rule of thumb we come across, cuts off more than one call in ten on every row of that table, because the 90th percentile sits above twice the median in each of them. Every one of those cut-offs becomes a retry that starts the clock again, on a call that was probably about to return — the worst possible moment to abandon it, since the provider has already done nearly all of the work it may bill for, and the retry pays for the same work again. Set the timeout from the tail of the exact task and model you are running, and set it after filtering stalls, or a handful of hung connections will push you toward a timeout that is far too generous for the honest tail.

The median hides a second thing: whether the model answered at all. Our quality scores are computed on the calls that came back, so a high score on a task where the model often fails to return a usable answer overstates what you get per call. Claude Opus 5 scores 8.91 out of 10 on labeling topic clusters when it answers, but answers on only 76% of attempts [^35]; Claude Haiku 4.5 scores 8.03 on the same task and answers on only 50% [^36][^35]. On generating research queries, Opus 5 scores 7.35 but answers on 70% of attempts [^37], and on structured content summarization Haiku 4.5’s 7.25 is measured on the 60% of attempts that returned [^10][^38]. Compare NVIDIA Nemotron-3 Nano 30B-A3B, which returns a usable answer, first time, on 99.3% of its 106,736 attempts [^39][^40][^41][^42]. For an interactive path, the latency that matters is the wait for an answer that actually arrives, retries included, and a 76% answer rate means roughly one call in four pays the wait at least twice.

The variant matters more than the family

Ask which model family is fastest and the data does not give a clean answer, because the spread inside a family is often larger than the spread between families. Generating a Reddit post takes 10.0 seconds at the median for GPT-5.6 Luna, 15 seconds for GPT-5.6 Terra and 30 seconds for GPT-5.6 Sol, each over 90 calls on the same inputs [^6][^4][^43]. That is a threefold difference between three variants sold under one name. Around them sit Claude Sonnet 5 at 32 seconds, Grok 4.5 at 38 seconds, Thinking Machines Inkling Small at 88 seconds, Moonshot Kimi K3 at 154 seconds and Qwen 3.7 Plus at 178 seconds [^6][^4][^43], so the choice between Luna and Sol matters about as much as the choice between the GPT-5.6 family and Claude.

The same pattern holds for generating an author voice profile, where Terra takes 16 seconds over 82 calls [^6][^4][^44][^45][^46] and Sol 28 seconds over 81 [^6][^4][^44][^46], while their judged quality on voice profiles is effectively tied: Terra 8.97 out of 10 (±0.14 after 45 judgments), Luna 9.05 (±0.15, 45) and Sol 9.13 (±0.13, 45) [^47][^48]. The slower variant is not buying you quality.

DeepSeek’s Flash and Pro tell the same story with a wider gap. Writing a short promotional social post takes 9.2 seconds at the median for V4 Flash over 2,371 calls [^6][^12][^18][^49][^50] and 31 seconds for V4 Pro over 277 [^6][^18][^50]. Adapting prompts takes 9.3 seconds for Flash, on thin evidence of 26 calls, against 31 seconds for Pro over 11,453 [^6][^27][^51]. Synthesizing titles for publication takes 13 seconds for Flash over 2,191 calls against 37 seconds for Pro over 577 [^6][^24][^52]. Translation takes 6.1 seconds for Flash, 24 seconds at the 90th percentile, over 5,709 calls, against 22 seconds for Pro over 140 [^6][^26][^32]. On extracting structured facts the two score identically, 9.54 out of 10 (±0.14 after 45 judgments) each [^53][^54].

Nor is newer slower, which is worth saying because the instinct to stay on last year’s variant “for speed” is common. Claude Sonnet 5 translates in 3.7 seconds at the median over 53 calls against 4.8 seconds for claude-sonnet-4-6 over 68 [^6][^26][^32], and triages engagement in 4.5 seconds over 261 calls against 6.0 seconds for claude-sonnet-4-6 over 180 [^6][^23][^55]. GPT-5.6 Luna translates in 3.3 seconds over 1,430 calls [^6][^26][^32][^44][^46] against 10 seconds for GPT-5.5 over 27 [^6][^26][^32].

What makes variant choice a free win is how often quality is a tie. On much production language work the top two answers are scored the same or nearly so. Detecting the language of a text is the extreme case: the top two answers scored exactly the same on 97.2% of 71 judged occasions and finished within half a point on 97.2%, a median gap of 0.0 on a 10-point scale [^6][^34]. Reviewing public engagement replies finished within half a point on 97.1% of 34 occasions [^56], writing a short promotional message on 96.8% of 31 [^57][^58], generating a publication title package on 96.0% of 25 [^59] and translating batches of text on 89.5% of 38 [^60]. Even for Reddit posts, where the top two agreed exactly on only 4.9% of 82 occasions, the median gap was 0.3 points [^61][^43], against a latency range from 10 seconds to 178 seconds. When the quality gap is a few tenths of a point and the latency gap is a factor of three to eighteen, latency is the decision.

The rule has one hard boundary: it is safe only per task. Both GPT-5.6 Terra and Sol collapse on suggesting content domains, to 3.87 (±0.2, 43 judgments) [^62][^63] and 3.84 (±0.1, 30) [^62][^63], the same two models that score 9.15 (±0.28, 71) [^64][^65][^66][^67] and 9.16 (±0.29, 43) [^65][^67] on scoring generated content for relevance. Claude Sonnet 5, the most broadly dependable model in our set, at 10.04 (±0.02, 125) on identifying languages [^9][^68] and 9.44 (±0.15, 35) on rewriting metadata paragraphs [^69][^70], scores 1.02 (±0.08, 30) on discovering themes [^71][^72]. A fast variant is a fast variant on the task you measured, and quality can fall off a cliff one task over.

What to do with this

Size waits from the 90th percentile of the exact task and variant, with stalls filtered out, and never from a median or a family-level number. Treat that timeout as a cost control as well as a latency control: the request you abandon may still be generated and invoiced, so an aggressive timeout on a long task can roughly double the spend per usable answer while your own token metering reports nothing, and the only place it surfaces is a reconciliation of invoiced spend against metered tokens. Then split the work into lanes by its shape. Narrow detection, triage and short writing answer in single-digit seconds at the median: language detection in 1.8 seconds for DeepSeek V4 Flash over 69,136 calls [^6][^32][^73][^34], engagement triage in 1.7 seconds for GPT-5.6 Terra over 201 [^6][^21][^22][^23][^74][^55][^75], a short promotional post in 1.5 seconds for Terra over 104 [^6][^18][^50]. That work belongs on a synchronous path with a timeout sized from its tail. Reference-heavy synthesis, filing analysis and long-form generation run from tens of seconds to minutes, as the Reddit-post spread above shows, and belong on a queue where a slow answer is a late answer rather than a failed one. Batch is a third lane with its own economics: GPT-5.4 Nano generates publication title packages at 8.04 out of 10 (±0.23) for $0.0015 per task run in batch mode [^76][^59], but we have no batch wall-clock figure, so batch is the answer only when the workflow tolerates queued completion.

Where two candidates are quality-tied on your task, pick by the measured median and tail on that task, and measure again when you switch variant, because the ordering is per task, not per brand. The live benchmark at https://llm-bench.kapualabs.com/ has the per-task latency and quality figures behind every number here, alongside batch and sync pricing, so you can set the quality bar you actually need and check whether the tail fits your interactive budget.