An AI agent can load the dataset, run valid R code, print a p-value, and still answer the scientific question incorrectly. The bug may be the statistical method it chose before the code ran.
P-Bench statistical test failures are designed to expose that gap. A new Stanford and University of Wisconsin–Madison preprint evaluates agents on 425 open-ended hypothesis-testing tasks. Each task gives the agent a scientific question, data, and a data description. The agent must choose a method, execute the analysis in R, report a p-value, and decide whether to reject or fail to reject the null hypothesis.
The paper also introduces Fisher-R1, a 14B open-weight agent trained on synthetic statistical tasks. The reported result is interesting. The more important contribution for builders is the evaluation design: executable code is necessary, but it does not validate the reasoning that selected the test.
P-Bench tests the decision before the p-value
Most code benchmarks reward a correct final output. Statistical analysis has an earlier failure point. A t-test, survival model, instrumental-variable estimator, or mixed-effects model may all run successfully while violating the assumptions of the question or data.
“The code executed” is evidence about the program. It is not evidence that the chosen analysis was appropriate.
P-Bench therefore scores both p-value accuracy and conclusion correctness. The paper says every answer key is tied to a logged execution of the canonical reference analysis and reviewed by domain experts. That creates a more inspectable target than a free-form model judgment.
The 425 tasks cover familiar and specialized methods
| Component | Count or requirement | Builder implication |
|---|---|---|
| Total tasks | 425 | Enough variety for diagnosis, but not a universal statistics certification. |
| Easy split | 203 | Common methods and cleaner task conditions. |
| Hard split | 222 | More demanding model specification, assumptions, or adversarial data-quality issues. |
| Method categories | 17 | Coverage ranges from common tests to Cox regression, IV/2SLS, Tobit, and mixed-effects models. |
| Agent output | Method, R execution, p-value, conclusion | A fluent narrative cannot replace an auditable analysis trace. |
The tasks draw from economics and biology papers plus biostatistics teaching material. Some include missing values or extreme and invalid observations. These perturbations test whether an agent notices the data problem instead of applying the first familiar method mechanically.
This is a useful complement to our analysis of OpenAI Astra’s mathematics proof package. A proof benchmark asks whether reasoning can establish a formal claim. P-Bench asks whether the agent can choose an empirical procedure whose assumptions fit the evidence.
Fisher-R1 is trained on executable synthetic work
The training pipeline starts with 8,642 synthetic tasks. A teacher model generated 4,611 trajectories that follow a five-step workflow: basic exploration, detailed exploration, assumption checking, method selection and analysis, then conclusion. Filtering retained 3,851 trajectories, about 83.5%, for supervised fine-tuning before reinforcement learning.
The authors report that Fisher-R1-14B improves single-trial success by 21% relative to DeepSeek-V4-Pro on average and by as much as 26% on the hardest tasks. These are author-reported results from the paper. They have not been independently reproduced in the public artifact available at publication time.
The repository does not yet support the open-weight claim
The linked Fisher-R1 repository currently contains only a README. It does not yet expose the P-Bench task files, training code, evaluation harness, or model weights described in the paper. The README says the release is coming soon.
That gap changes how the result should be used. The paper is enough to evaluate the proposed methodology and reported numbers. It is not enough to reproduce the leaderboard, inspect individual failures, or verify that the released model matches the evaluated checkpoint.
Artifact availability should be part of benchmark scoring. Our Qwen agent benchmark analysis reached the same practical conclusion from a different angle: ranking claims become more useful when the evaluator can inspect the tasks, harness, and variance—not only the final position.
A safer statistical agent needs four separate gates
- Question gate: require the agent to define the estimand, null hypothesis, unit of analysis, and data-generating assumptions.
- Method gate: make it list plausible tests, reject unsuitable ones, and explain the choice before execution.
- Execution gate: run the code in a controlled environment and preserve the script, package versions, warnings, and output.
- Interpretation gate: check whether the conclusion follows from the computed evidence, including uncertainty and limitations.
Do not combine those gates into one pass/fail score. A system that chooses the right test but makes a coding error needs a different repair from one that executes flawless code for the wrong model. The trace should make that distinction visible.
P-Bench still leaves important scientific workflows outside the frame
- Each task evaluates one hypothesis test, not a multi-stage analysis pipeline.
- The benchmark does not yet test multiple-comparison correction across a family of searched hypotheses.
- Private clinical or scientific datasets can introduce governance, consent, and distribution-shift problems absent from a benchmark.
- A p-value and binary conclusion do not capture effect size, practical significance, causal identification quality, or study design flaws.
- The public repository is incomplete, so independent reproduction must wait.
The paper explicitly positions Fisher-R1 and P-Bench as evaluation and oversight tools, not substitutes for human statistical review. That boundary is especially important in medicine. A high benchmark score does not authorize an agent to make clinical decisions or validate a study without qualified review.
My verdict: audit method choice before admiring the output
P-Bench points at a failure that polished data agents can hide. The chart renders, the code runs, and the report sounds scientific. None of those facts proves that the selected statistical test fits the question.
Copy the benchmark’s decomposition into your own workflow. Score method selection, execution, numerical output, and conclusion separately. Preserve the full trace. Then rerun the same analysis with a statistician-designed control set before giving the agent consequential authority.
Read the primary sources
- Read the Fisher-R1 and P-Bench preprint.
- Inspect the current Fisher-R1 repository and check whether the promised artifacts have arrived.
Which part of your analysis workflow can fail while every line of code still executes?
Checked August 10, 2026. Task counts, training counts, benchmark design, and comparative results come from the authors’ August 7 preprint. Repository contents were inspected directly and contained only a README at publication time.
