A new benchmark called Taste-Bench asks a different question about AI agents: before the outcome is visible, can the model choose the better direction at a real decision fork? The first results suggest that current frontier models still make many expensive long-horizon choices poorly.
What Taste-Bench means by taste
Taste-Bench AI agents are evaluated on judgment rather than only final task success. The authors define taste as the ability to choose a productive direction during a long task, such as which hypothesis to test, which implementation to continue or which failed path to abandon.
Each question freezes a recorded trajectory at a fork. The model sees the task, the work completed so far and two candidate next steps. It must choose before seeing the later evidence that reveals which path produced the better result.
Where the 502 questions come from
The released benchmark contains 502 questions. Of those, 390 come from software-engineering trajectories and 112 from machine-learning research. The engineering material draws from SWE-bench and SWE-bench Pro rollouts. The research material uses public METR MALT transcripts from RE-Bench and HCAST runs.
The authors mine two types of fork. Parallel forks compare independent attempts that diverged from a similar state. Detour forks identify a path an agent followed, later abandoned after failure and then recovered from inside the same run.
Why random guessing scores 25, not 50
Every question is asked twice, once with the published option order and once with the options reversed. A question counts as correct only when the model picks the right decision in both orders. Random guessing therefore succeeds one quarter of the time. A model that always selects the first position scores zero under this rule.
This paired-order protocol reduces position bias and makes the 59.7 percent top reported score easier to interpret. It is not equivalent to ordinary two-choice accuracy.
The reported leaderboard
| Reported result | Value | Interpretation |
|---|---|---|
| Released questions | 502 | A small but concrete set of real decision forks |
| Engineering questions | 390 | SWE-bench and SWE-bench Pro trajectories |
| Research questions | 112 | METR MALT research trajectories |
| Random paired-order score | 25 percent | Both option orders must be correct |
| Best reported average | 59.7 percent | Substantial headroom remains on this benchmark |
The repository lists GPT-5.6 Sol at 59.7 percent and GPT-5.5 at 59.5 percent, followed by Claude Opus 5 at 55.5 percent under the published protocol. These are author-reported results from August 2026, not independent benchmark replications.
Later evidence appears to create myopia
The paper reports that decision forks become much harder when the evidence that distinguishes the better choice appears later in the trajectory. Increasing the reasoning budget did not fix the pattern. That finding supports the authors’ argument that more thinking tokens are not automatically better judgment.
A model can produce a locally plausible step while missing the downstream constraint that makes the step expensive. In production, that failure may appear as a clean patch that creates review debt, a research plan that exhausts the budget or a sequence of tool calls that reaches the wrong objective efficiently.
The training result is promising and preliminary
The authors also report distilling hindsight judgment from a teacher that could see the outcome into a student model. They say this improved the student’s benchmark choices and raised success on held-out SWE-bench Pro tasks. The result connects local decision quality with an end-to-end outcome, which is the benchmark’s most important claim.
It still needs independent replication. Dataset construction, teacher quality, task selection and contamination controls can all affect the result. The gated dataset is intended to reduce training contamination, while the evaluation code is available under an MIT license.
How teams can use the idea without overclaiming
- Record decision points in agent runs, not only final outputs.
- Save the alternatives considered and the evidence available at the time.
- Review costly detours and successful recoveries as separate events.
- Test whether a model keeps the same choice after option order changes.
- Measure budget spent after a poor fork and reviewer time required to recover.
- Use hindsight to improve policy while preventing test examples from entering training data.
This complements end-to-end reports such as the 10,000-agent proof review and our Grok visualizer verification, where an impressive outcome still requires evidence about the process and claim boundary.
The benchmark limitations matter
Taste-Bench is a new preprint and an author-created benchmark. Its 502 questions cover two technical domains and rely on mined trajectories whose outcomes label the preferred branch. The filters remove trivial or disputed examples, but that process may favor forks that are easier to formalize after the fact.
The score does not prove that a model lacks judgment everywhere, and it should not be used as a single ranking for general intelligence. It offers a focused test of one capability that ordinary pass/fail benchmarks often hide.
The practical verdict
Taste-Bench is valuable because it evaluates a decision before hindsight is available and ties that decision to recorded downstream evidence. The strict paired-order scoring and released harness make the work testable.
The headline result is not that agents have no taste. It is that even strong models leave substantial room for better long-horizon judgment, and simply increasing the reasoning budget did not remove the reported weakness.
Primary sources
- arXiv: The Tasteful Agent
- GitHub: Taste-Bench code and leaderboard
- Hugging Face: gated Taste-Bench dataset card
Checked September 23, 2026. Taste-Bench is a September 22 preprint. Scores and training gains are author-reported and require independent replication.
