An agent that remembers is not automatically an agent that learned. The memory has to improve the next clean run.
PAST-Bench agent memory is built around that distinction. The new preprint evaluates agents across fresh sessions, clears conversational context, and runs matched controls with persistence switched on or off. Its strongest idea is not a leaderboard. It is the requirement that saved experience change later behavior under the same task conditions.
The authors report 26 scenarios and 204 episodes across seven models and four agent frameworks. They evaluate four steps: saving useful information, retrieving it later, applying it to the task, and updating it when new evidence arrives.
Fresh-session controls make the benchmark useful
Many memory demos leave the original chat history, tool trace, or hidden process state available. A model may appear to remember when it is simply reading context that never went away. PAST-Bench clears the session and compares the same scenario with persistence enabled and disabled.
The relevant score is not “did the agent store something?” It is “did the stored experience improve a later clean attempt?”
This matched design makes the result easier to interpret. A positive difference can be attributed more plausibly to the persistence mechanism, although ordinary model variance and framework effects still remain.
The benchmark tests a full memory loop
| Capability | Question | Common failure |
|---|---|---|
| Save | Did the agent preserve useful experience? | Storing noise or omitting the decisive detail |
| Retrieve | Did the right memory return in a fresh session? | Relevant evidence exists but is never surfaced |
| Apply | Did the memory change the plan or answer correctly? | Citing a memory without using it |
| Update | Can later evidence correct or replace it? | Stale advice persists after conditions change |
That last stage matters most in production. An agent that perfectly retrieves an obsolete deployment procedure can be worse than one that forgets it. Memory quality includes deletion, conflict resolution, provenance, and time.
The reported gains are promising, not uniform proof
The paper reports positive overall persistence deltas for four open models: GLM-5.1 at +0.20, Kimi K2.6 at +0.17, DeepSeek-V4-Pro at +0.17, and MiniMax-M2.7 at +0.13. Those numbers come from the authors’ evaluation and should not be read as universal gains across every agent task.
The sample spans multiple models and frameworks, which helps. It still reflects 26 designed scenarios, the authors’ scoring rules, and an August 4 preprint that has not completed peer review. Real products should reproduce the persistence-on versus persistence-off comparison on their own failures.
Hermes+ shows why mechanism evidence matters
The researchers also tested a modified memory layer called Hermes+. They report that its mechanism-evidence score increased from 0.64 to 0.73 and its overall mean moved from 0.13 to 0.15. The paper then adds the caveat that the overall difference is smaller than run-to-run variation.
That is responsible reporting. Better evidence that a memory was retrieved and used does not automatically establish a stable task-level improvement. A production evaluation should score both: outcome quality and a trace showing which memory changed the decision.
This does not prove recursive self-improvement
PAST-Bench evaluates persistence across episodes. It does not show an agent autonomously rewriting its core model, generating open-ended capabilities, or improving without limit. The narrower claim is more useful: a structured memory layer can help some agents carry task-relevant experience into a new session.
That framing fits our review of Meta Muse Code’s restartable handoffs. Continuity is valuable only when the next operator or agent can inspect what survived and why.
Update: memory needs revocation, not only retrieval
Update, August 10, 2026: A separate preprint called TEPA sharpens the “update” stage in PAST-Bench. Measuring whether memory helps a clean session is only half the job. A deployed memory system also needs an explicit way to mark older evidence invalid when newer evidence contradicts it.
TEPA represents observations as keyed precedents with a validity state. When fresh evidence conflicts under the same key, the older precedent is revoked from ordinary retrieval but preserved in history for audit. That separates deletion from invalidation: the agent stops seeing stale evidence by default without erasing the record of what changed.
| Memory policy | Controlled drift | Executable file test |
|---|---|---|
| Append-only | 0.210 | 0.203 |
| Last-write-wins | 0.210 | 0.203 |
| No memory | 0.309 | 0.298 |
| TEPA | 0.950 | 0.950 |
In the authors’ 50-seed reversal test, append-only and last-write-wins memory fell below the no-memory control after the hidden rule changed. The file-backed test, which uses real CSV and JSON operations, reproduced the pattern. TEPA’s 0.950 result reflects the same scheduled reversal and deterministic memory-update logic in both experiments. These are author-reported preprint results; no public code repository or independent reproduction is linked.
The scope is narrower than “solved memory.” TEPA matches a strong last-write-wins cache on clean single-hop MemoryAgentBench, but the paper reports sharp failures on multi-hop and very-long-context settings. Its theoretical argument is limited to stale conflicts where an old and new value share one key. Revocation does not construct a missing retrieval chain or find the right evidence inside an enormous context.
- Give every stored fact a stable conflict key and provenance.
- Separate active, revoked, and pending states instead of treating memory as an append-only list.
- Remove revoked facts from normal retrieval while preserving an audit trail.
- Require fresh evidence or a validated trial before promoting a replacement.
- Test deliberate reversals alongside ordinary save-and-retrieve tasks.
Read the TEPA preprint. Its useful production lesson is simple: a memory is not trustworthy merely because it has provenance. It also needs a current validity state and a tested path out of retrieval.
A production memory test should start with matched runs
- Collect real tasks that previously failed because context was missing, stale, or contradictory.
- Start every test from a fresh session with identical tools, permissions, and inputs.
- Run one condition without persistent memory and one with it.
- Record the exact memory retrieved, its source, age, and effect on the plan.
- Score task success, harmful regressions, latency, token cost, and reviewer time.
- Add a changed-world test that requires the agent to update or reject an old memory.
- Repeat enough times to separate a stable gain from ordinary run variance.
Cross-session messaging can create another hidden memory channel. Our Claude Code cross-session messaging guide explains why identity and permissions must be part of the test. Long-running hosted agents add a second issue: our Managed Deep Agents checklist covers state, replay, and recovery.
My verdict: copy the control, not the headline
PAST-Bench is a useful evaluation proposal because it makes persistence compete against a no-persistence control under clean sessions. Its reported improvements justify more testing. They do not justify turning on an opaque memory store for every workflow.
Start where repetition is expensive and the correct experience can be audited. Require provenance, expiry, correction, and deletion. Then measure whether memory improves outcomes often enough to pay for the new failure surface.
Read the paper and repository
- Read the PAST-Bench preprint on arXiv.
- Inspect the authors’ PAST-Bench repository.
What would your agent need to remember to beat a clean no-memory run?
Checked August 10, 2026. Scenario counts, model and framework counts, reported deltas, and Hermes+ results come from the authors’ August 4 preprint and repository.
