An agent that remembers is not automatically an agent that learned. The memory has to improve the next clean run.
PAST-Bench agent memory is built around that distinction. The new preprint evaluates agents across fresh sessions, clears conversational context, and runs matched controls with persistence switched on or off. Its strongest idea is not a leaderboard. It is the requirement that saved experience change later behavior under the same task conditions.
The authors report 26 scenarios and 204 episodes across seven models and four agent frameworks. They evaluate four steps: saving useful information, retrieving it later, applying it to the task, and updating it when new evidence arrives.
Fresh-session controls make the benchmark useful
Many memory demos leave the original chat history, tool trace, or hidden process state available. A model may appear to remember when it is simply reading context that never went away. PAST-Bench clears the session and compares the same scenario with persistence enabled and disabled.
The relevant score is not “did the agent store something?” It is “did the stored experience improve a later clean attempt?”
This matched design makes the result easier to interpret. A positive difference can be attributed more plausibly to the persistence mechanism, although ordinary model variance and framework effects still remain.
The benchmark tests a full memory loop
| Capability | Question | Common failure |
|---|---|---|
| Save | Did the agent preserve useful experience? | Storing noise or omitting the decisive detail |
| Retrieve | Did the right memory return in a fresh session? | Relevant evidence exists but is never surfaced |
| Apply | Did the memory change the plan or answer correctly? | Citing a memory without using it |
| Update | Can later evidence correct or replace it? | Stale advice persists after conditions change |
That last stage matters most in production. An agent that perfectly retrieves an obsolete deployment procedure can be worse than one that forgets it. Memory quality includes deletion, conflict resolution, provenance, and time.
The reported gains are promising, not uniform proof
The paper reports positive overall persistence deltas for four open models: GLM-5.1 at +0.20, Kimi K2.6 at +0.17, DeepSeek-V4-Pro at +0.17, and MiniMax-M2.7 at +0.13. Those numbers come from the authors’ evaluation and should not be read as universal gains across every agent task.
The sample spans multiple models and frameworks, which helps. It still reflects 26 designed scenarios, the authors’ scoring rules, and an August 4 preprint that has not completed peer review. Real products should reproduce the persistence-on versus persistence-off comparison on their own failures.
Hermes+ shows why mechanism evidence matters
The researchers also tested a modified memory layer called Hermes+. They report that its mechanism-evidence score increased from 0.64 to 0.73 and its overall mean moved from 0.13 to 0.15. The paper then adds the caveat that the overall difference is smaller than run-to-run variation.
That is responsible reporting. Better evidence that a memory was retrieved and used does not automatically establish a stable task-level improvement. A production evaluation should score both: outcome quality and a trace showing which memory changed the decision.
This does not prove recursive self-improvement
PAST-Bench evaluates persistence across episodes. It does not show an agent autonomously rewriting its core model, generating open-ended capabilities, or improving without limit. The narrower claim is more useful: a structured memory layer can help some agents carry task-relevant experience into a new session.
That framing fits our review of Meta Muse Code’s restartable handoffs. Continuity is valuable only when the next operator or agent can inspect what survived and why.
A production memory test should start with matched runs
- Collect real tasks that previously failed because context was missing, stale, or contradictory.
- Start every test from a fresh session with identical tools, permissions, and inputs.
- Run one condition without persistent memory and one with it.
- Record the exact memory retrieved, its source, age, and effect on the plan.
- Score task success, harmful regressions, latency, token cost, and reviewer time.
- Add a changed-world test that requires the agent to update or reject an old memory.
- Repeat enough times to separate a stable gain from ordinary run variance.
Cross-session messaging can create another hidden memory channel. Our Claude Code cross-session messaging guide explains why identity and permissions must be part of the test. Long-running hosted agents add a second issue: our Managed Deep Agents checklist covers state, replay, and recovery.
My verdict: copy the control, not the headline
PAST-Bench is a useful evaluation proposal because it makes persistence compete against a no-persistence control under clean sessions. Its reported improvements justify more testing. They do not justify turning on an opaque memory store for every workflow.
Start where repetition is expensive and the correct experience can be audited. Require provenance, expiry, correction, and deletion. Then measure whether memory improves outcomes often enough to pay for the new failure surface.
Read the paper and repository
- Read the PAST-Bench preprint on arXiv.
- Inspect the authors’ PAST-Bench repository.
What would your agent need to remember to beat a clean no-memory run?
Checked August 10, 2026. Scenario counts, model and framework counts, reported deltas, and Hermes+ results come from the authors’ August 4 preprint and repository.
