Set LLM Timeouts Per Task, and Size Them on the 90th Percentile

The task, not the model, decides how long an LLM call takes: the same model that answers in a few seconds on language detection needs minutes on topic matching. And within any one task, the slowest tenth of calls run several times longer than the median. A single global timeout sized for the typical call is wrong in both directions, and fixing it is a config change, not a model change.

The spread is a task spread

Detecting language takes DeepSeek V4 Flash 1.8 seconds at the median, measured over 69,136 synchronous calls under a 600-second per-call deadline [^1][^2][^3][^4]. Reference-preserving analytical writing takes Tencent Hy4 Preview 325 seconds at the median over 29 calls [^5][^1][^6]. That is the range the benchmark covers, from a couple of seconds to more than five minutes, and no one deadline serves both ends.

It would be easy to read that as fast models versus slow models, but the spread lives inside a single model. Moonshot Kimi K3 detects language in 4.3 seconds at the median over 93 calls [^1][^2], and the same model takes 210 seconds at the median to match clients to topics over 37 calls [^5][^1][^7]. One model, one client, and a wait measured in minutes rather than seconds depending only on what you asked.

So the timeout belongs on the task, not in a shared HTTP client. Identification and triage fit interactive budgets of a few seconds; extraction, scoring and short generation need tens of seconds; matching, taxonomy work and reference-preserving writing belong on background queues with minute-scale deadlines. If your client library has one timeout constant, that constant is currently either killing your long tasks or letting your short ones hang.

Within a family, the small model is the fast one

The task sets the tier, and the model variant moves you within it. DeepSeek V4 Flash against DeepSeek V4 Pro on matched work shows how much room there is.

Flash is lower on both, by enough to move the work from a tens-of-seconds budget toward an interactive one. If the smaller model clears your quality bar for that task, the latency alone is a reason to pick it, and the per-task timeout you set should follow the variant you actually deploy.

What the median hides

Per-task budgets fix the first problem, but a budget set at the task’s median still fails on a predictable fraction of traffic. NVIDIA Nemotron-3 Ultra 550B answers relevance scoring for a topic report in 2.6 seconds at the median but 18 seconds at the 90th percentile, over 121 calls [^1][^14][^15]. Qwen 3.7 Plus on the same work is 20 seconds at the median but 96 seconds at the 90th percentile, over 560 calls [^1][^14][^15].

That tail is not a slow-model problem either. DeepSeek V4 Flash scores the same relevance task in 4.4 seconds at the median but 19 seconds at the 90th percentile, over 1,839 calls [^1][^14][^15]. Being quick on the typical call does not make a model evenly quick, and a fast median is exactly the number that tempts people into a tight deadline.

The practical consequence is that a median-sized timeout strands one call in ten, and those are the calls you then retry, paying the wait twice and adding load to the same provider. Size the enforced deadline on the task’s 90th percentile for the model you deploy, keep the median as the figure you show users or put in an SLA, and log the two separately so you notice when the tail drifts before the median does.

Where to look next

The per-task latency figures quoted here come from the live benchmark at https://llm-bench.kapualabs.com/, where you can set the quality bar you actually need before comparing models on the tasks that resemble yours. The method behind the numbers is documented at https://llm-bench.kapualabs.com/methodology/, and each task page on the site, such as the relevance-scoring and language-detection pages cited below, carries the per-model breakdown.