Length and latency belong to the task, not the model

Picking a model from a leaderboard bakes in two assumptions: that “better” means the same thing across your workload, and that speed is a property of the model. The latest llm-bench snapshot undercuts both. On the same benchmark, judges rewarded longer answers on one task and shorter answers on its nearest neighbour. And a single model’s median response time ran from under a second to more than a minute depending only on what it was asked to do. Both findings point the same way: prompts, routing and timeouts have to be set per task, not per model.

Judges did not reward length. They rewarded the right length for the task.

The usual worry with LLM-as-judge scoring is verbosity bias: longer answers look more thorough, so they win. The benchmark records the average output length of the answers judges picked against the ones they passed over, and the picture is more specific than that. Length helps on some tasks and hurts on others, and the direction flips between tasks that look almost interchangeable.

Generating research queries went to shorter answers, 1,617 output tokens for the winners against 1,908 for the rest [^1]. Validating those same queries went the other way, 1,565 against 1,375 [^2]. Vetting subreddits favoured longer answers at 937 against 713 tokens [^3][^4], while selecting subreddits favoured shorter ones at 1,250 against 1,571 [^5][^6]. Generating a section of a document rewarded brevity, 1,118 against 1,434 [^7][^8]; generating the prompts for those sections rewarded the opposite, 8,209 against 5,960 [^9]. Translation went to shorter answers at 1,150 against 1,699 [^10][^11], summarization to longer ones at 3,333 against 2,203 [^12][^13][^14].

The gaps can be large in either direction. On suggesting content domains the winning answers averaged 2,257 tokens against 574 for the rest [^15][^16][^17], and on extracting atomic fact claims 6,299 against 2,926 [^18]. On assigning sections to topic clusters the winners were less than half the length of the losers, 332 against 692 [^19][^20], and on language detection 58 against 268 [^21][^22].

Two things follow. First, a quality win has to be checked against length before you read it as a capability lead. A model that happens to run verbose will win the verbose tasks and lose the concise ones, and neither result says much about how well it reasons. Second, a “be concise” or “be thorough” instruction copied across a pipeline is a bet that every task wants the same thing, and the data says that bet loses on a large share of tasks.

One model, from under a second to three minutes

The second finding is about time. All latency figures here are medians over synchronous calls with a 600-second per-call deadline, and the spread inside a single model is as wide as the spread between models on a single task.

Thinking Machines Inkling Small detects language in 609 ms at the median over 144 calls [^21][^22][^23]. The same model takes 88 seconds to generate a Reddit post [^24][^21][^25] and 91 seconds to repair newlines in Markdown [^24][^21][^26]. Moonshot Kimi K3 runs from 4.3 seconds on language detection [^21][^22] to 210 seconds on matching clients to topics [^24][^21][^27]. Tencent Hy4 Preview runs from 6.7 seconds [^21][^28][^23] to 325 seconds on reference-preserving analytical writing [^24][^21][^29].

Fast models show the same shape at a smaller scale. GPT-5.6 Terra detects language in 913 ms [^21][^22] and writes a short promotional post in 1.5 seconds over 104 calls [^21][^30][^31], but takes 16 seconds to generate an author voice profile [^24][^21][^32][^33][^34]. Claude Sonnet 5 assigns sections to topic clusters in 1.7 seconds [^24][^21][^20] and takes 48 seconds for analyst writing with referenced claims [^21][^35][^36].

Hold the task fixed and the model choice still matters, sometimes by an order of magnitude. Writing a Reddit post takes GPT-5.6 Luna 10.0 seconds at the median and GPT-5.6 Sol 30 seconds, each over 90 calls [^24][^21][^25]; Claude Sonnet 5 takes 32 seconds, Grok 4.5 38 seconds, and Qwen 3.7 Plus 178 seconds [^24][^21][^25]. Selecting a portfolio of social posts takes Claude Opus 5 2.2 seconds over 35 calls [^37] and Grok 4.6 84 seconds [^24][^21][^37]. So a per-task latency budget is not one number either; it is a number per model per task.

The median is not the timeout

Where the benchmark reports the 90th percentile as well as the median, the two often disagree by a factor of four or more. DeepSeek V4 Flash scores document relevance for a topic report in 4.4 seconds at the median but 19 seconds at the 90th percentile, over 1,839 calls [^21][^38][^39]. The same model translates in 6.1 seconds at the median and 24 seconds at the 90th percentile, over 5,709 calls [^21][^10][^11]. NVIDIA Nemotron-3 Ultra 550B scores relevance in 2.6 seconds at the median and 18 seconds at the 90th percentile [^21][^38][^39]. Qwen 3.7 Plus summarizes content in 51 seconds at the median and 179 seconds at the 90th percentile [^12][^21][^14].

A timeout set from the median on any of these would fail one call in ten. Where the tail is measured, size timeouts and user-facing expectations from it; where it is not, assume a multiple of the median rather than the median itself.

What to do with this

Split the workload by task before choosing models. Language detection, triage, selection and short promotional copy sit comfortably in a synchronous, user-facing path on the fast models above. Reference-preserving writing, taxonomy and client matching, long-form generation and filing analysis belong in an asynchronous or batched path with allowances measured in minutes, whichever model runs them. Set the length instruction per task rather than per pipeline, and when a model wins a task, check whether it won on length before promoting it. Treat comparisons resting on a few dozen calls as provisional until more data lands.

Where to look next

Every task in this post has its own page on the live benchmark, with the judged scores, the winner-versus-loser output lengths, and the median and tail latencies per model. Start at https://llm-bench.kapualabs.com/ to set the quality bar your production task needs, then open the per-task pages (for example https://fronset.ai/benchmark/task/research-query-generation/) to see which length and which latency budget your own tasks actually want.