Fronset Benchmark

Inference Latency Benchmarking

In the benchmark, median response time depends more on the kind of work than on the model: Muse Spark 1.1 takes 5.4 seconds on language detection [^1][^2] but 69 seconds on topic client matching [^1][^3][^4][^5]. So task type is the first test of which LLM features can run while a user waits and which go to background jobs. On a given task, though, the model can move a feature across that line.

Three tiers, set by the kind of work

Every response time quoted here is a median over synchronous calls, where the caller waits for each answer, under a 600-second per-call deadline. The median is the time within which half of the calls finished. None of these figures measures output quality or cost, so they are a guide to turnaround only.

The quick tier is classification, triage, language identification and short scoring or selection, which return in seconds on most models. GPT-5.4 Nano returns language detection in 860 ms [^1][^2], and Claude Sonnet 5 chooses an X post in 2.0 seconds [^1][^3][^6].

A middle band, covering most other tasks including labelling, short writing and the lighter topic steps, falls between roughly 5 and 40 seconds. Muse Spark 1.3 Contributor writes report executive summaries in 31 seconds [^1][^3][^7][^8], and Muse Spark 1.2 returns content domain suggestions in 16 seconds [^9].

The slow tier is long, structured or referenced output, which runs to tens of seconds or minutes. MiniMax M3 triages engagement in 2.3 seconds [^1][^10][^11] but takes 106 seconds on trading recommendations [^1][^3][^12]. Qwen 3.7 Plus takes 244 seconds on structured output extraction over 31 calls [^1][^3][^13].

One workflow can span two tiers

Topic work splits into a fast step and slower ones, even on a single model. Gemini 3.5 Flash assigns sections to existing topic clusters in 4.4 seconds over 45 calls [^1][^3][^14]. Topic discovery clustering, which builds the clusters themselves, takes it 51 seconds over 28 calls [^1][^3][^15], and client matching 47 seconds over 33 calls [^1][^3][^4][^5].

Client matching is slower still on other models: 68 seconds on Inkling Small [^1][^3][^4][^5] and 168 seconds on Qwen 3.7 Plus [^1][^3][^5]. Section assignment also varies by model: Qwen 3.8 Flash takes 7.4 seconds against 17 seconds for Qwen 3.8 Max, both over 29 calls [^1][^3][^16].

Where the model does make the difference

Where models ran the same task, the differences are often large. On structured extraction, GPT-5.6 Luna answers in 14 seconds, GPT-5.6 Terra in 18 seconds and GPT-5.6 Sol in 44 seconds, each over 44 calls [^1][^3][^13]. The three share a task and a deadline, so the gap reflects the models; whether their outputs are equally accurate is not established.

On some tasks the gap between models spans tiers. On social post portfolio selection, Claude Opus 5 answers in 2.2 seconds over 35 calls [^17] and Grok 4.6 in 84 seconds over 30 calls [^1][^3][^17]. On reference-preserving analytical writing, GPT-6 Sol takes 20 seconds over 26 calls [^18] against 325 seconds for Tencent Hy4 Preview [^1][^3][^18], the longest median here.

Two limits: task labels and call counts

Several similar tasks carry different labels. Muse Spark 1.2 triages engagement opportunities in 6.1 seconds [^19] and Muse Spark 1.1 triages engagement in 15 seconds [^1][^10][^11], but whether the two labels describe identical work is not established. Unless two models appear side by side on one task, a gap between them is a gap between two measured tasks.

Many medians also rest on small call counts, such as the 25 calls behind Nemotron-3 Ultra 550B’s 24 seconds on topic discovery clustering [^1][^3][^15][^20][^21].

What to do with this

  • Sort features by the kind of work first. Classification, triage, language identification and short scoring are candidates for the path a user waits on; long, structured or referenced output belongs in background jobs.
  • Split multi-step features by step. In topic work, section assignment suits interactive use on Gemini 3.5 Flash or Qwen 3.8 Flash, while discovery and client matching belong in background jobs on every model measured here.
  • For work a user waits on, choose the fastest model measured on that exact task, not on a similarly named one, and test its slowest calls before setting a latency budget.

To check a specific model and task, see the live benchmark at https://fronset.ai/benchmark/. The method is documented at https://fronset.ai/benchmark/methodology/.