Best LLMs for Profile Pool Matching
Matches a source object to the most suitable candidates in a supplied profile pool using caller-defined dimensions, eligibility rules, thresholds, and creation policy. When permitted, it may propose a profile for a material coverage gap without claiming that a real candidate exis
Run this task on Fronset — request an invitation
Models
Frontier on this task: Tencent Hy4 Preview at 8.70 / 10. Quality bar at 90%: 7.83.
point-estimate floor (CI low) · upper CI (less certain) · Bars sorted by blended cost; best-value model first. Greyed rows are MEDIUM+ models whose point estimate does not clear the bar.
| Model | Quality score | CI low | Cost / 1k runs | vs best value |
|---|---|---|---|---|
| Xiaomi MiMo V2.6 Flash | 8.17 / 10 | 7.78 | $0.67 | best value |
| Claude Haiku 5.5 | 8.28 / 10 | 7.96 | $1.25 | 1.9x more expensive |
| Qwen 3.7 Plus | 7.86 / 10 | 7.46 | $6.47 | 9.7x more expensive |
| Thinking Machines Inkling Small | 7.85 / 10 | 7.47 | $8.73 | 13x more expensive |
| GLM-5.3 Flash | 8.62 / 10 | 8.28 | $9.44 | 14x more expensive |
| Claude Sonnet 5.5 | 8.25 / 10 | 7.86 | $23.04 | 35x more expensive |
| Meta Muse Spark 1.3 | 8.50 / 10 | 8.09 | $30.93 | 46x more expensive |
| Tencent Hy4 Preview | 8.70 / 10 | 8.39 | $31.89 | 48x more expensive |
| Thinking Machines Inkling | 8.07 / 10 | 7.60 | $33.90 | 51x more expensive |
| Claude Opus 5.5 | 8.24 / 10 | 7.88 | $47.05 | 71x more expensive |
| Grok 4.7 | 7.86 / 10 | 7.41 | $66.71 | 100x more expensive |
| GLM-5.3 | 8.34 / 10 | 7.90 | $104.56 | 157x more expensive |
| Tencent Hy3 | 7.44 / 10 | 7.10 | $2.87 | 4.3x more expensive |
| NVIDIA Nemotron-3 Ultra 550B | 7.08 / 10 | 6.60 | $12.33 | 18x more expensive |
| MiniMax M3 | 7.73 / 10 | 7.50 | $5.38 | 8.1x more expensive |
| Qwen 3.8 Max | 7.81 / 10 | 7.31 | $61.54 | 92x more expensive |
| DeepSeek V4.1 Flash | 7.71 / 10 | 7.23 | $3.06 | 4.6x more expensive |
| Gemini 3.5 Flash Lite | 6.88 / 10 | 6.47 | $2.21 | 3.3x more expensive |
Cost breakdown
| Model | Quality | Confidence | Cost / 1k runs | Overpay | Mode |
|---|---|---|---|---|---|
| Xiaomi MiMo V2.6 Flash ★ Xiaomi MiMo | 8.17 / 10 CI [7.78, 8.55] | MEDIUM | $0.67 | best value | batch |
| Claude Haiku 5.5 Anthropic | 8.28 / 10 CI [7.96, 8.61] | MEDIUM | $1.25 | 1.9x | batch |
| Qwen 3.7 Plus Alibaba Cloud (DashScope) | 7.86 / 10 CI [7.46, 8.27] | MEDIUM | $6.47 | 9.7x | batch |
| Thinking Machines Inkling Small OpenRouter | 7.85 / 10 CI [7.47, 8.23] | MEDIUM | $8.73 | 13x | batch |
| GLM-5.3 Flash Z.AI | 8.62 / 10 CI [8.28, 8.96] | MEDIUM | $9.44 | 14x | batch |
| Claude Sonnet 5.5 Anthropic | 8.25 / 10 CI [7.86, 8.63] | MEDIUM | $23.04 | 35x | batch |
| Meta Muse Spark 1.3 OpenRouter | 8.50 / 10 CI [8.09, 8.92] | MEDIUM | $30.93 | 46x | batch |
| Tencent Hy4 Preview best OpenRouter | 8.70 / 10 CI [8.39, 9.01] | MEDIUM | $31.89 | 48x | batch |
| Thinking Machines Inkling OpenRouter | 8.07 / 10 CI [7.60, 8.54] | MEDIUM | $33.90 | 51x | batch |
| Claude Opus 5.5 Anthropic | 8.24 / 10 CI [7.88, 8.61] | MEDIUM | $47.05 | 71x | batch |
| Grok 4.7 xAI | 7.86 / 10 CI [7.41, 8.31] | MEDIUM | $66.71 | 100x | batch |
| GLM-5.3 Z.AI | 8.34 / 10 CI [7.90, 8.78] | MEDIUM | $104.56 | 157x | batch |
Overpay shows how much more you pay than the best-value model that clears the quality bar (marked ★) — the best-value good-enough option. "16x" means you overpay 16× — 16× that reference for no quality benefit above the bar. Typical call shape for this task: 14737 input tokens → 651 output tokens, EMA-tracked from production traffic. Cost is the observed, all-in $ per 1,000 task runs: each model's own measured usage on this task — output verbosity, thinking/reasoning tokens, cache reads and writes, and the spend on its billed failures — priced at current list rates and adjusted by the billing overhead we actually reconcile against provider invoices. Models that answer tersely cost what they actually cost; models that think at length pay for it. Not comparable to providers' advertised $/1M list rates — this is what running the task costs, not a per-token price.
Evaluation rubric
Judge candidate fit under the supplied dimensions, eligibility compliance, evidence for each match, ranking and threshold calibration, appropriate reuse of the existing pool, correct unmatched behavior, and restraint and usefulness when proposing a coverage-gap profile. Penalize superficial keyword matching, popularity or identity bias, invented attributes, and invented real candidates. Output cardinality, identifier validity, and schema conformance are deterministic.
Output schema
Every answer on this task is checked against this JSON Schema, whichever model wrote it. An answer that doesn't fit counts as a model failure, and the call is retried on another model.
{
"$defs": {
"NewAuthorData": {
"description": "Data for creating a new fictional author.\nUsed when the LLM decides to create a new author rather than assign to existing.",
"properties": {
"biography": {
"description": "Short bio under 200 chars (e.g., 'Clinical researcher specializing in women's health with 15 years of industry experience.')",
"maxLength": 200,
"title": "Biography",
"type": "string"
},
"content_domains": {
"anyOf": [
{
"items": {
"type": "string"
},
"type": "array"
},
{
"type": "null"
}
],
"description": "Names of content domains this author belongs to. Must be a subset of the template's content domains. Empty = generalist.",
"title": "Content Domains"
},
"content_themes": {
"description": "Keywords for future matching (5-10 keywords)",
"items": {
"type": "string"
},
"title": "Content Themes",
"type": "array"
},
"expertise_areas": {
"description": "List of expertise areas (3-5 areas)",
"items": {
"type": "string"
},
"title": "Expertise Areas",
"type": "array"
},
"gender": {
"anyOf": [
{
"type": "string"
},
{
"type": "null"
}
],
"default": null,
"description": "Portrait gender for image generation: 'male', 'female', or 'neutral'. Use the historical figure's gender (e.g. 'male' for Adam Smith, 'female' for Marie Curie). Omit if uncertain.",
"title": "Gender"
},
"inspiration_note": {
"anyOf": [
{
"type": "string"
},
{
"type": "null"
}
],
"default": null,
"description": "Internal note: who/what inspired this persona",
"title": "Inspiration Note"
},
"interests": {
"description": "List of interests that humanize the author",
"items": {
"type": "string"
},
"title": "Interests",
"type": "array"
},
"name": {
"description": "Author's display name (e.g., 'Dr. Elara Stanton')",
"title": "Name",
"type": "string"
},
"writing_style": {
"description": "Writing style: 'academic', 'conversational', 'investigative', 'analytical', or 'accessible'",
"title": "Writing Style",
"type": "string"
}
},
"required": [
"name",
"biography",
"expertise_areas",
"content_themes",
"writing_style"
],
"title": "NewAuthorData",
"type": "object"
}
},
"description": "Output model for LLM-based author matching.\nThe LLM either assigns content to an existing author or creates a new one.",
"properties": {
"action": {
"description": "Either 'assign' to use existing author or 'create' to make a new one",
"title": "Action",
"type": "string"
},
"author_id": {
"anyOf": [
{
"type": "string"
},
{
"type": "null"
}
],
"default": null,
"description": "UUID of the existing author if action is 'assign'",
"title": "Author Id"
},
"new_author_data": {
"anyOf": [
{
"$ref": "#/$defs/NewAuthorData"
},
{
"type": "null"
}
],
"default": null,
"description": "New author details if action is 'create'"
}
},
"required": [
"action"
],
"title": "AuthorMatchingResult",
"type": "object"
}Prompt templates
The published template for this task.
default ·
LLMB_PROFILE_POOL_MATCHING_SYSTEM +
LLMB_PROFILE_POOL_MATCHING_USER
(858 calls in window)
System prompt
Compare the requirements and relevant attributes of source_object with the candidates in profile_pool. Apply matching_profile exactly for candidate type, eligibility, comparison dimensions, weights, thresholds, ranking, result count, and gap policy. Preserve candidate identifiers and support every match with evidence from both the source object and candidate profile. Reuse qualifying candidates before proposing a new profile. If no candidate qualifies and profile proposals are disabled, return the configured unmatched or coverage-gap result. If proposals are enabled, describe the missing profile requirements without inventing a real person, organization, asset, or other existing entity, and without implying that any real entity holds the described views or produced the described work. Where matching_profile requires the proposed profile to be named after a real figure, follow that naming rule exactly — a named persona is a described role, not a claim about that figure. Do not match on name recognition, protected characteristics, popularity, or unsupported attributes unless the supplied policy explicitly makes an attribute relevant and lawful. Treat empty optional values as absent and return only the requested result. Your response must conform exactly to this output schema: {schema_json_string}.
User prompt
Inputs — source_object: {source_object}; profile_pool: {profile_pool}; matching_profile: {matching_profile}. Use only these inputs to complete the task defined by the system prompt.