The winning agent did not discover one brilliant optimizer trick and stop. It kept working for 8.7 days, ran 811 experiments, and spent the second half of its improvement after the first 24 hours.
Prime Intellect’s new Fable 5 NanoGPT Speedrun result is useful because the site exposes more than a rank. It shows the harness, effort setting, step record, 24-hour checkpoint, token totals, experiment count, calls, duration, and 41 curated traces across the broader study.
Fable 5 paired with Claude Code at high effort reached 2,726 training steps. Prime Intellect says that closes 81.7% of the gap between a 3,290-step baseline and the 2,600-step human record. The study contains 153 autonomous runs across 18 frontier models, with additional runs still marked in progress on August 24.
The task freezes most of the system
The benchmark asks an agent to reduce the number of training steps needed to reach a fixed nanoGPT validation-loss target. Data, model architecture, batch size, sequence length, validation procedure, and trial seeds are frozen. The agent can change the optimizer, hyperparameters, learning-rate schedule, and initialization.
Each score is the mean of eight fixed validation trials, and the rules prohibit selecting only a favorable seed. Network access is disabled. One reported step represents one forward-backward pass, which makes the target more legible than a benchmark based on an opaque judge.
That controlled setup does not make the result general. It measures search over one compact training problem. A useful optimizer change here may not survive a larger model, different data, mixed precision, distributed training, or a production reliability constraint.
Fable 5 needed time to earn the headline score
| System | 24-hour record | Best record | Days | Experiments |
|---|---|---|---|---|
| Fable 5 + Claude Code high | 3,010 | 2,726 | 8.7 | 811 |
| Opus 5 + Claude Code max | 3,045 | 2,920 | 2.9 | 292 |
| Kimi K3 + prime-agent max | 3,125 | 2,930 | 3.6 | Not reported |
| GPT-5.6 Sol + Codex xhigh | 3,160 | 3,042 | 6.1 | 963 |
The baseline-to-Fable improvement is 564 steps: 3,290 minus 2,726. At 24 hours the improvement was 280 steps. The later 284-step gain therefore represents 50.4% of the final improvement over baseline. More than half of the result arrived after the comparison window many teams would use for an agent bake-off.
This does not mean every long run will keep improving. It means a fixed 24-hour cutoff answers a different question from an unconstrained record attempt. Prime Intellect provides an equal-budget view so readers can compare the systems at the same agent-hour, experiment, or output-token budget.
The benchmark ranks model, harness, and effort together
The table is not a clean Fable-versus-Opus-versus-GPT model ranking. Fable 5 ran through Claude Code at high effort. Kimi K3 appears with two different harnesses. GPT-5.6 Sol ran through Codex at xhigh. Search behavior, tool interface, retry policy, context handling, and subagent use all affect what the model can discover.
This is the same measurement problem we saw in DarwinX evolving an agent harness around a frozen model. When the wrapper changes the result, reporting only the model name hides the mechanism a builder can actually reproduce.
The result also complicates simple token comparisons. Prime Intellect reports 800 million total tokens and 1.1 million output tokens for the Fable run, versus 2.9 billion total tokens and 2.2 million output tokens for GPT-5.6 Sol. Those totals reflect different interaction patterns and prompt reuse. They are workload measurements, not a direct bill without provider prices, caching rules, and tool-call accounting.
The traces matter more than another screenshot
Prime Intellect says 41 full trajectories are available, including tool calls, subagents, and scratchpads. That is where a team should look for transferable behavior: whether the agent isolates variables, preserves a working baseline, detects a noisy result, abandons dead ends, and records enough state to reproduce a later run.
A leaderboard can tell you who ended lower. A trace can tell you whether the search process is acceptable. An agent that occasionally finds a strong configuration after hundreds of untracked edits is less useful than one that produces a modest gain with a clean experiment ledger and a rollback path.
Our LLMRouter analysis makes a related point: routing is valuable when it matches a task and budget, not when it turns one aggregate score into a universal winner.
A practical autonomous-research acceptance test
- Freeze the evaluation. Lock data, seeds, metric code, and the baseline before the agent starts.
- Price the search budget. Set maximum wall time, calls, tokens, experiments, and compute, then compare all systems at that boundary.
- Require a ledger. Every experiment needs a hypothesis, exact change, result, artifact, and link to the parent run.
- Re-run the winner. Validate the final configuration outside the agent’s search history and on a held-out condition.
- Inspect failure behavior. Confirm the agent stops on corrupted metrics, exhausted budget, unstable training, or an unauthorized file change.
- Keep the human baseline visible. The target is not merely to beat another agent; it is to reduce the cost of reaching a result a competent person can trust.
My verdict: this is a search-system benchmark
Fable 5 produced the best reported record in this study, and it did so with a large, sustained search. That is evidence that the model-harness-effort combination can run a productive optimization campaign under the benchmark’s rules.
It is not evidence that Fable 5 is the best coding model for every repository or the cheapest research agent. I would use the public traces to design a smaller internal contest: one real optimization task, one fixed budget, one held-out validation, and a requirement that another engineer can reproduce the winning change.
Read the benchmark record
- Explore the NanoGPT Speedrun Frontier results and equal-budget view.
- Read the frozen task rules and scoring procedure.
- Inspect the experiment repository and released trajectories.
Would your agent still look impressive if every system received the same wall time and experiment budget?
Checked August 24, 2026. Run counts, scores, budgets, tokens, calls, and durations are Prime Intellect’s published measurements. The 50.4% figure is Musthave.AI’s calculation: 284 post-day-one steps divided by the 564-step total improvement over baseline.
