Skip to content

Benchmarks

These are the platform’s own eval results, published as they come out of the release run. Every number here can be checked: each workflow links to the cases that produced it, each run links to the commit it measured, and one command runs the same benchmark on your own deployment.

The cases are synthetic by design. Every prompt, document and expected answer was written for the benchmark, so the results carry no user’s data and nothing is withheld, including the cases a model fails.

Three measurements, for each model and each workflow:

  • Quality. Cases passing out of cases run. Each case runs three times, and a case passes when most of its samples pass. Failing cases are listed by name.
  • Speed. Median time for the whole task, every turn of it, as the deployed runtime measured it, and the median time to the first streamed token.
  • Cost. Mean dollars per task, as the platform metered the model calls at its catalog rates, passing and failing samples alike.
Every model, the whole golden set
ModelCases passingCost per taskMedian task timeMedian first tokenRun
claude-haiku-5-5142 of 143 (k=3)$0.00066.8 s2.7 s2026-11-13, v1.35.0, fc17f5007
claude-sonnet-5140 of 143 (k=3)$0.0106.3 s1.5 s2026-11-13, v1.35.0, 4dd1bb313
claude-opus-5-5138 of 143 (k=3)$0.0227.4 s2.2 s2026-11-13, v1.35.0, fc17f5007
By workflow: cases passing, cost per task and median task time
Workflowclaude-haiku-5-5claude-sonnet-5claude-opus-5-5
Tool selection9/9 · $0.0005 · 5.1 s9/9 · $0.0095 · 5.0 s8/9 · $0.019 · 6.7 s
Document Q&A24/24 · $0.0007 · 4.7 s24/24 · $0.0089 · 5.5 s22/24 · $0.020 · 7.4 s
Attachments: tools12/12 · $0.0006 · 8.7 s12/12 · $0.012 · 7.4 s12/12 · $0.020 · 8.5 s
Project knowledge base13/13 · $0.0006 · 6.9 s13/13 · $0.0084 · 4.4 s13/13 · $0.028 · 5.9 s
Custom agents10/10 · $0.0004 · 4.3 s10/10 · $0.0083 · 3.3 s10/10 · $0.014 · 4.4 s
Project memory9/10 · $0.0007 · 9.7 s10/10 · $0.0078 · 7.4 s10/10 · $0.015 · 8.4 s
Skills: outcome10/10 · $0.0011 · 10.1 s9/10 · $0.016 · 9.0 s10/10 · $0.050 · 12.9 s
Terminal client · advisory6/6 · $0.0005 · 3.3 s6/6 · $0.0084 · 3.5 s6/6 · $0.015 · 5.1 s
Costly workflows · advisory10/10 · $0.0010 · 12.4 s9/10 · $0.018 · 14.1 s10/10 · $0.033 · 16.0 s
Long conversations8/8 · $0.0007 · 10.2 s7/8 · $0.014 · 12.1 s8/8 · $0.024 · 13.6 s
Instruction following9/9 · $0.0001 · 2.4 s9/9 · $0.0022 · 2.7 s9/9 · $0.0033 · 2.5 s
Refusals8/8 · $0.0004 · 5.4 s8/8 · $0.0074 · 8.3 s6/8 · $0.013 · 7.4 s
Skills: selection8/8 · $0.0008 · 10.0 s8/8 · $0.013 · 9.2 s8/8 · $0.031 · 11.5 s
General chat · judged6/6 · $0.0004 · 5.0 s6/6 · $0.0073 · 8.2 s6/6 · $0.017 · 9.3 s

Every failing case

A case fails when most of its samples fail. Each links to the file that defines what was asked and what counts as a pass.

Measured on a development deployment, AWS us-west-2, AgentCore Runtime. Generated 2026-11-13 byeval_harness.py publish; the raw document isbenchmarks.json.

  • Families are small. Most have 8 to 24 cases, so one case is several points. The counts are shown rather than percentages for that reason.
  • “Costly workflows” reproduces the usage patterns that carry the most cost in production: revising a document over several turns, fetching pages, and analysing spreadsheets. Its cases also cap how many tool calls a task may take, so a model that loops fails there even when its final answer is right.
  • Advisory families report without gating a release; judged ones are scored by a model rather than a fixed check. See Methodology for what each measurement does and does not tell you.
  • Prices are the development deployment’s. It runs Regional (us.*) inference profiles, about 10% above the Global ones a production deployment can use. Tokens and times compare across the two; dollars scale.