A cheap token can produce an expensive failure. AWS has published a benchmark that compares OpenAI models by cost per correct answer and completed agent trajectory, a more useful buying frame than price per million tokens alone.
AWS published its OpenAI model cost benchmark on September 11, 2026 and released the benchmark code on GitHub. Results below are AWS-reported and the comparison has an important configuration asymmetry.
The benchmark changes the denominator
Price-per-token tables assume output quality is interchangeable. AWS instead evaluates cost per correct answer, cost for an agent trajectory and the quality of a generated deliverable. That connects model spend to something a team can accept or reject.
| AWS-reported result | GPT-5.6 Sol | GPT-5.4 Mini |
|---|---|---|
| AIME accuracy | 75% | 37% |
| GPQA accuracy | 68% | 43% |
| MMLU-Pro accuracy | 82% | 59% |
Luna’s AIME result shows why sticker price can mislead
AWS reports that GPT-5.6 Luna cost $0.0021 per correct AIME answer, compared with $0.0139 for GPT-5.4 Mini after a July price change. Dividing by successful outcomes reverses the intuitive assumption that the cheaper-looking model must be cheaper to use.
That arithmetic is valuable, but it belongs to this dataset and setup. Your prompts, tool calls, retry policy, latency requirements and acceptance tests can produce a different winner.
The model settings are not matched
AWS says the Bedrock-hosted GPT-5.6 models were run with reasoning disabled, while the OpenAI API baselines used their default settings. That makes the published result useful as a reproducible workload snapshot, but not a clean intrinsic comparison of model capability.
The evaluations also use sample sizes ranging from 48 to 198. Large gaps deserve attention; narrow gaps should be treated as directional until repeated. Region, service tier and current pricing must be checked before a purchasing decision.
A fair internal rerun needs matched controls
- Pin the exact benchmark commit and record every model identifier.
- Match reasoning effort where the platforms expose equivalent controls.
- Run each task several times and save failures, retries and tool traces.
- Use a blinded acceptance rule for generated deliverables.
- Add latency, human review and failed-run cost to token charges.
- Recalculate with the price and region your production account will use.
Our AI agent cost-control guide shows how to cap experiments before routing real work. The Cognition SWE-2 benchmark analysis explains the same danger from another angle: a model can look excellent on one task family and weak on another.
Cost per correct answer is still incomplete
A benchmark answer can be automatically scored. Production work often cannot. A code change needs tests and review. A research memo needs source verification. A customer action may need policy approval. Define the accepted unit before comparing models.
For agents, a useful denominator is cost per accepted task after human review. Include model charges, tool infrastructure, retries, reviewer time and rollback. That prevents a low token bill from hiding expensive cleanup.
My take: reproduce the ranking, keep the method
The benchmark’s most durable contribution is not which model wins. It is the decision to price successful outcomes instead of raw tokens. Teams should copy that method with their own tasks and reject any universal ranking that ignores configuration.
Read and rerun the primary material
Checked September 12, 2026. Scores and costs are attributed to AWS. No independent MustHave.ai rerun is claimed.
