A benchmark can look beautifully clean while measuring the wrong contest. In this case, the winning system saw the answer sheet repeatedly and was allowed to run past the clock.
The headline result was strong enough to stop asking questions
Orcetra, a small AutoML engine, appeared to beat FLAML and AutoGluon on 57.1% of 513 OpenML datasets at a nominal 60-second budget. Against FLAML alone at 30 seconds, the reported win rate was 78.4%. Those are the sort of numbers that turn into a README badge and a buying decision.
Then the authors audited their own system. Their new paper, Winning by Peeking, says the margin came from protocol defects rather than a proven search advantage. This is an author-reported correction, not an independent replication, but it comes with per-dataset results and regeneration scripts.
The search kept looking at the answer sheet
The first problem was model selection. Orcetra fitted candidate pipelines on the training split, scored each one on the test split, and kept the highest test score it had seen. The competing frameworks selected internally from training data and touched the test split once.
That makes the comparison asymmetric. A test set is supposed to estimate how the chosen model will perform on unseen data. Once the search process consults it dozens of times, the test set becomes part of the optimization loop. Some of the apparent gain will be real signal; some will be luck that the search learned to select.
The practical rule is simple: choose on training or validation data, then score the final choice on the test set once.
The 60-second race often lasted two minutes
The second problem was the clock. Orcetra checked whether time remained before launching a candidate, but it did not stop a model already fitting. A job launched at second 59 could run to completion.
The logs show the consequence. At a nominal 60-second budget, Orcetra used a median 120 seconds per dataset. FLAML used 70 seconds and AutoGluon 62. Orcetra exceeded its stated limit on 78% of datasets and doubled it on half of them. A shared parameter name did not create a shared resource limit.
What changed after the protocol was repaired
| Measure | Original or replayed protocol | Corrected protocol |
|---|---|---|
| Model selection | Repeatedly used the test split | Used a validation split; test scored once |
| Time enforcement | Checked between candidates | External deadline with a fixed grace period |
| Orcetra win rate on corrected-run subset | 59.4% | 34.3% |
| Pairwise advantage | Appeared favorable | No significant difference from FLAML or AutoGluon |
The corrected run moved selection to validation data, evaluated the test split once, enforced the deadline from outside the search process, and pinned each framework to an equal share of the machine. Orcetra’s win rate on that subset fell by 25.1 percentage points, from 59.4% to 34.3%.
The authors attribute 4.8 points of the decline to the selection rule and most of the remaining difference to unequal compute. That breakdown is useful because it stops us from blaming every inflated result on test-set leakage. Resource control was the larger distortion here.
The paper is unusually honest, but it is not the final word
I like this paper because the authors did something rare: they published the number that made their system look good, showed why it was misleading, and reran the experiment. That is more valuable than quietly replacing a chart.
Still, the corrected result has limits. It uses one 80/20 split rather than repeated folds, the 513 datasets are not a curated benchmark suite, the corrected run covers a stratified subset, and AutoGluon did not include its fastai component. The work was also run on one system and one hardware platform. Treat 34.3% as the corrected result under these conditions, not a universal ranking.
A five-minute audit before you trust an AutoML benchmark
- Find the selection split. The search, tuner, ensemble, and early-stopping logic must not choose against the final test set.
- Find the real clock. Ask for realized wall time per framework, not only the budget argument passed to each API.
- Check resource isolation. CPU cores, memory pressure, GPU access, and concurrent workers must be comparable.
- Inspect run provenance. Result files need run IDs, code versions, machine state, and a clear rule for superseded runs.
- Demand paired traces. Per-dataset outcomes and confidence tests matter more than one aggregate win rate.
This same discipline applies beyond AutoML. Our review of P-Bench shows how an AI can execute code correctly yet choose the wrong statistical test. FinRank makes a similar point about financial answers that look right while resting on the wrong evidence.
My verdict: audit the harness before the leaderboard
A leaderboard row cannot tell you whether the system peeked, overspent, or inherited a friendlier machine. The harness can. Before adopting the winner, reproduce one slice with an untouched test set and a deadline enforced outside every competitor.
Read the evidence
- Read the paper and its experimental HTML.
- Compare the public claim still shown on the Orcetra product page.
- Use our guide to choosing an AI model without losing a weekend to benchmarks.
What is the first benchmark check you run before letting a winning model into production?
