A corrected SQL query is useful once. It becomes memory only when it improves a later question.
A new paper on Text-to-SQL memory separates those two outcomes. The researchers keep the single-shot solver fixed, store verified repair episodes, and ask whether that stored experience helps an unseen question from the same database.
On the BIRD benchmark, the authors report that stored verified corrections improved held-out first-attempt accuracy by 4.34 percentage points. They calculate that this recovers 44.4% of the headroom provided by running on-demand repair on those same questions.
The useful result is not “memory works.” It is narrower: verified, database-specific repairs can transfer part of their value to different questions, and the evaluation has to measure that transfer separately from simple replay.
Repair and memory solve different problems
Test-time scaling spends more compute after an initial SQL attempt. A verifier identifies an error, a repair loop tries again, and a difficult question may eventually succeed. If the corrected query is discarded, the system pays that cost again when a related problem returns.
A memory layer stores something from the successful repair and retrieves it before the next attempt. The next query should improve on its first try, before another repair loop starts. That is a stronger standard than showing that the system can replay a stored answer to an identical question.
Do not score reusable memory by how often it recalls the past. Score it by how often it improves a different future task.
The evaluation separates four numbers
- Single-shot baseline: accuracy before repair or retrieved memory.
- On-demand repair: the additional accuracy available when the system spends more compute on the current question.
- Exact replay: whether stored experience answers a recurring question again.
- Held-out transfer: whether memory improves the first attempt on a different question from the same database.
The paper also measures cross-question retention, which checks whether knowledge from one repaired problem persists when the surface question changes. Keeping these outcomes separate prevents a system that memorizes benchmark prompts from looking like one that learned reusable database structure.
The reported 4.34-point gain needs the right denominator
The authors report that a strong open-weight model rises from roughly 62% execution accuracy to roughly 72% with up to three execution-guided repair rounds. Stored verified corrected queries then improve held-out first-attempt accuracy by 4.34 percentage points.
The paper’s 44.4% figure is a descriptive ratio: the held-out memory gain divided by the headroom that on-demand repair creates on the same questions. It is not the probability that any remembered query will solve the next task, and it should not be reported as a universal memory success rate.
All of these figures are author-reported from an August 7 preprint. They support the evaluation design and justify reproduction. They do not establish the same gains for another database mix, model provider, schema scale, or production query distribution.
Database-specific content did most of the work
The controlled interventions point toward a practical mechanism. Memory helps when it exposes the solver to the target database: real identifiers, values, joins, and recurring query structures. The paper reports that even shuffling the pairing between stored questions and SQL can preserve useful signal, suggesting that database exposure matters more than copying a semantically perfect example.
Reliable verification also matters. A memory writer should not store a repair merely because a model produced it confidently. The query needs an execution or correctness signal strong enough to prevent a plausible mistake from becoming a reusable precedent.
Broader retrieval coverage produced supported gains in the authors’ tests. Richer memory-card formats, more elaborate retrievers, and repair-specific content did not show statistically supported advantages. That is a useful warning against adding an expensive memory framework before proving that simple verified examples are being retrieved at all.
The public artifact is useful but not a one-click replication
The authors released an MIT-licensed repository with the single-shot solver, repair loop, memory writing, retrieval and evaluation code, tests, aggregated results, a frozen run manifest, and scripts to regenerate publication artifacts.
That is stronger than a paper with no implementation. It is still important to separate artifact regeneration from full experimental replication. The repository does not include the full raw model-call archive, API credentials, private endpoints, or the complete BIRD dataset. A full rerun requires benchmark access and a compatible OpenAI-style endpoint, and provider changes can prevent bit-identical results.
A production memory policy needs write controls
- Verify before writing. Store only repairs backed by execution, tests, or another dependable correctness signal.
- Scope by database and schema version. Do not let a join pattern from one customer silently cross into another.
- Store provenance. Keep the source question, generated SQL, corrected SQL, verifier result, model version, and timestamp.
- Measure transfer separately. Report exact replay, related-question retention, and held-out first-attempt improvement as different metrics.
- Test retrieval coverage. A good example that never appears cannot help.
- Expire on schema change. Rename, quarantine, or reverify memories when tables, columns, permissions, or business definitions change.
- Keep a no-memory control. Re-run matched traffic without retrieval to prove that the store still earns its cost.
This complements our review of PAST-Bench agent memory. PAST-Bench asks whether persistence improves a clean later session across agent tasks. The Text-to-SQL study drills into a narrower mechanism: whether verified repair episodes transfer value to new questions inside the same database.
Hosted agents add another operational layer. Our Managed Deep Agents checklist covers state, replay, and recovery when long-running jobs depend on stored context.
My verdict: treat memory as a verified database asset
The paper’s strongest contribution is its decomposition. Exact replay, repair success, and future transfer are not interchangeable. A system can look excellent on recurring questions while doing nothing for unseen ones.
Start with verified corrected queries, broad enough retrieval, strict database boundaries, and a held-out transfer test. Do not buy a richer memory format until the simple version proves that it improves the next first attempt.
Read the paper and repository
- Read the Text-to-SQL memory preprint on arXiv.
- Inspect the authors’ reproducibility repository on GitHub.
What would your memory layer have to improve on a different question before you called it reusable?
Checked August 10, 2026. Benchmark design, reported gains, mechanism interventions, and artifact descriptions come from the authors’ August 7 preprint and public repository. All performance figures are author-reported.
