Fronset Benchmark

Quality and Cost Benchmarking

In the benchmark, Grok 4.5 scores 8.94 out of 10 on topic sequence optimization at $0.01 per task run [^1][^2], and 4.66 on content domain suggestion at the same $0.01 [^3][^4]. Price says little about the quality a model delivers, and some spend buys nothing usable at all. Model choice and budgets both have to be worked out task by task.

Cheaper models can tie pricier ones

The benchmark’s current results cover 27 models on 54 tasks. Each score comes with a ± interval, and where two models’ intervals overlap, they are tied on quality, however far apart their prices are.

On public engagement reply review, Claude Opus 4.8 scores 8.4 out of 10 (±0.31) at $0.02 per task run in sync mode, meaning a synchronous call rather than batch mode [^5][^6]. Tencent Hy3 scores 8.14 (±0.46) at $0.0003 per run [^5][^6]. Opus 4.8’s higher score is not a lead: the intervals overlap, so the two are tied, and Hy3 delivers that quality at a fraction of the price.

Paying more does not guarantee a higher score, either. Grok 4.7 scores 7.42 (±0.38) at $0.02 per run [^6]. That puts it below Claude Opus 4.8, and it cannot be separated from Tencent Hy3 on quality while costing far more per run.

On content-to-image prompt generation (all costs in sync mode), GPT-5.6 Sol scores 9.13 (±0.14) at $0.03 per task run, repriced at current rates, and Thinking Machines Inkling scores 9.07 (±0.26) at $0.0096, measured on its own usage [^7][^8]. The two are tied, and Inkling’s quoted cost is lower. GLM-5.3 scores 8.86 (±0.14) at $0.05 per run, the most expensive run in the group, without scoring above the top two [^8].

A top result on one task need not carry over to another. Inkling, tied at the top on image prompts, scores 4.33 (±0.44) on content domain suggestion [^3][^4], the task where every model measured here scores lowest.

Some spend buys output that cannot be parsed

A price per run also does not show how much of the output you can use. On several tasks, a share of billed output could not be parsed.

On research community selection, $5.03 of the $6.75 spent (74.5%) produced unusable output [^9][^10]. Generating visual theme configurations lost $5.66 of $8.38 (67.5%) [^11][^12], and query generation lost $5.59 of $9.71 (57.6%) [^13][^14].

Parse failure varies widely across tasks: at the low end it was 8.3%, on profiled document section analysis [^15][^16]. Whether the cause is the model, the prompt or the parser is not established.

What to do with this

  • Pick a model per task, from its scores on that task rather than its price or its results elsewhere.
  • Read overlapping ± intervals as ties, and choose between tied models on cost.
  • Where a model’s output feeds a later step, measure usable output per dollar alongside price per token.

Check the figures against the bar your work needs at https://fronset.ai/benchmark/, where you can set your own quality threshold. The method is documented at https://fronset.ai/benchmark/methodology/.