Suppose you want to know whether your AI agent should try three approaches in parallel or one at a time. Whether it should give up after four retries or eight. Whether it should go wide first and deep later.
These are strategy questions, and they are brutally expensive to answer. Judging one solution is cheap — you run it and score it. Judging a strategy means letting it steer an entire run to the end, across dozens or hundreds of attempts, before you learn anything. Then you change one knob and pay for the whole thing again.
A paper published in September 2026 makes a simple observation that changes the economics: you already paid for this information. The run you finished last week recorded every decision it made and what each one produced. That record is not just history. It is an environment you can act inside.
TL;DR: A finished AI agent run is a free simulator, because every decision in it is stored next to the outcome it produced. Dream-RSI (Google, Google DeepMind, UMD, UVA — September 2026) uses this to test thousands of strategies at zero execution cost, reporting up to 162× fewer agent calls on one benchmark. Build agents on a record you own →
What Is a Replay Simulator?
A replay simulator is a completed run treated as an environment. The defining property is that every outcome is already stored, so evaluating a different strategy requires reading rather than executing.
The paper's own analogy is navigation. An agent crossing unfamiliar terrain takes bad routes, hits dead ends, backtracks, and gradually builds a map. Once that map exists, planning a better route does not require physically walking every street again. You reason over what you recorded.
The crucial word is already. Nobody builds the simulator. It accumulates as a by-product of doing the work, and it costs nothing extra to have.
Why Is Improving an Agent's Strategy So Expensive?
Because meta-level feedback is delayed and the strategy space is enormous — two walls at the same time.
The paper states the dilemma directly. A fixed strategy cannot learn from the experience it accumulates, so it keeps spending on directions that have already failed. But optimizing the strategy online means each candidate needs a full rollout before it returns any signal, and most candidates are bad.
| Fixed strategy | Online strategy optimization | Replay-based | |
|---|---|---|---|
| Learns from experience | No | Yes | Yes |
| Cost per candidate tested | n/a | One full rollout | One read |
| Candidates testable per real run | 0 | ~1 | Thousands |
| Limited to recorded territory | n/a | No | Yes |
That last row is the honest trade, and we come back to it.
How Does the Dream-RSI Loop Work?
It alternates between acting in the world and evaluating inside the record, in three stages.
Stage 1 — Online exploration. The current strategy guides a coding agent through real work. Every attempt becomes a node: it resumes a parent's saved workspace, produces something, gets evaluated, and stores its score alongside the artifact and diagnostics.
Stage 2 — Build the simulator. The completed tree becomes a replay world. Nothing is trained and nothing is approximated. The tree is the model.
Stage 3 — Improve by replay. Many candidate strategies are generated and each is scored against the recorded trees. A strategy here makes exactly the decisions a strategy makes: which branch to continue, how many attempts to run in parallel, when to stop. Because every outcome is stored, the score is a lookup.
The Guarantee Worth Stealing
The selection step has a property most teams should copy outright: the incumbent strategy is included in the candidate set.
Because the next strategy is chosen as the highest scorer among all evaluated versions, and the current one is among them, the selected strategy is by construction no worse than the incumbent on the recorded history. A tuning loop that cannot regress on its own benchmark is a very different object from one that can.
The caveat matters too: that is a guarantee about the offline score on a fixed history, not a promise about future online behavior.
What Results Does the Paper Report?
Substantially lower discovery cost at comparable or better quality, across three domains and eight tasks.
The clearest single picture is the Lasso regularization-path task, measured in cumulative discovery-agent calls:
| Domain | Benchmark | Reported result |
|---|---|---|
| Algorithm engineering | Lasso regularization path | 317 agent calls vs 550 for the fixed-exploration control, beating sklearn and glmnet on all six held-out datasets |
| Mathematical optimization | Sum-difference, circle packing, autocorrelation | Matches or surpasses strong baselines within under 1,000 generations |
| GPU kernels | KernelBench VGG16, LayerNorm, ConvDiv, ConvMax | 2.43× and 1.79× fewer generations, or 2.09× and 1.44× higher performance at comparable budget |
There is also a behavioral finding that reads as almost biological. Tracking the learned strategy across rounds on one kernel task, the paper observes it conserving effort as performance improves — dropping from 110 evaluated attempts to 50 — then spending again when progress plateaus, climbing back to around 92 and coinciding with further gains.
Nobody wrote that pattern down. It emerged from scoring strategies against what had already happened.
Read the Numbers in Three Tiers
This matters, because the "162×" figure travelled much further than its context did. There are three different tiers of claim here and they deserve different levels of confidence:
| Tier | What it says | Confidence |
|---|---|---|
| The abstract | "competitive or improved discovery quality while substantially reducing discovery cost in several settings" — no digit at all | Highest. This is the claim the authors chose to lead with |
| The paper body | Up to 162× fewer agent calls than SimpleTES, 1.7× over the fixed-exploration control, 1.79×–2.43× fewer generations on kernels | Solid, but each is a best case on one task against one baseline |
| Social media | "162×" as a headline capability | Weakest. True of one comparison, not a general property |
The 162× comes from comparing 317 agent calls against a prior system reported at 51,200 — a real result on the Lasso task, and not a claim about how much faster the method is in general. The paper's math results are also mixed rather than a clean sweep: it matches the best reported circle-packing figure and improves sum-difference, while a competing system retains the best autocorrelation result, albeit at more than fifty times the generation budget.
Did Injecting History Into the Prompt Help?
No. It made things worse, which is the paper's most quotable negative result.
The obvious alternative to a replay simulator is to summarize past runs into high-level guidance and paste it into the prompt. The paper tested exactly that and reports:
"explicit directional guidance consistently underperforms its unguided counterpart across both paradigms under equivalent discovery budgets"
Their interpretation is that strong semantic bias about where to look next over-constrains the search space and impedes diverse exploration.
We reached the same direction independently, in a completely different setting — app generation rather than automated discovery — and wrote it up in Structure Beats Instruction. Two teams, unrelated problems, same wall: history left as structure beats history injected as guidance.
Is This Really Recursive Self-Improvement?
Not in the sense most people mean by that phrase, and the paper's own framing invited the argument.
When the paper hit Hacker News it drew a sharp and largely correct objection: nothing about the model changes. The weights are fixed. The coding agent is fixed. The evaluator is fixed. What gets rewritten between rounds is the code of the exploration strategy — the orchestration layer that decides which branch to continue, how many attempts to run at once, and when to stop.
That is a real and useful thing to automate. It is not a system bootstrapping its own intelligence, and it does not compound without bound.
| The popular reading | What the paper actually does |
|---|---|
| "AI improves itself" | A fixed model rewrites one orchestration component |
| Model capabilities grow | Model weights and capabilities are unchanged |
| Unbounded compounding | Bounded by what the recorded history contains |
| A new kind of intelligence | A cheaper way to tune search strategy |
We think the honest framing is the most interesting one anyway. The valuable idea here is not self-improvement. It is that you have been throwing away a free simulator. Every completed run is an asset that can answer strategy questions you would otherwise pay to re-ask, and almost nobody stores their work in a shape that allows it.
Where Did This Idea Come From?
It is a new application of a well-established family, and the lineage is worth knowing because it tells you what to expect.
The model-based reinforcement learning line — learn from simulated experience:
| Year | Work | Contribution |
|---|---|---|
| 1990 | Sutton, Dyna | Learning, planning and acting in one loop, where planning runs against a model instead of the world |
| 2018 | Ha & Schmidhuber, World Models | Train a compact model of the environment, then train the policy entirely inside its "dream" |
| 2019–2025 | Hafner et al., Dreamer line | Improve a policy by backpropagating through imagined latent rollouts |
The LLM discovery-agent line — evolve programs against an evaluator:
| Year | Work | Contribution |
|---|---|---|
| 2025 | AlphaEvolve | Evolutionary coding agent that found a new 4×4 matrix-multiplication algorithm |
| 2025 | ShinkaEvolve | Sample-efficient program evolution — a circle-packing record in ~150 samples |
| 2025 | Darwin Gödel Machine | An agent that rewrites its own codebase, keeping only empirically validated changes |
| 2026 | SimpleTES | Evaluation-driven scaling — the 51,200-generation baseline Dream-RSI measures against |
| 2026 | Dream-RSI | Applies the replay idea at the meta level, where the "model of the environment" is literal recorded history rather than a learned approximation |
The adjacent agent-memory line — reuse experience as context:
| Year | Work | Contribution |
|---|---|---|
| 2025 | ReasoningBank | Distil successes and failures into reusable reasoning strategies |
| 2025 | Agentic Context Engineering | Grow a context playbook by incremental structured updates |
| 2026 | Dream-RSI | Argues history is worth more as a simulator than as retrieved context — see the prompt-guidance result above |
LEARNED WORLD MODEL REPLAY SIMULATOR
─────────────────── ─────────────────
approximates dynamics stores exact outcomes
can imagine unseen branches cannot leave the record
must be trained accumulates for free
error compounds over horizon no model error at all
generalizes beyond logged data covers visited space (exactly)
but degrades out of distribution ▲ trade coverage for exactness — and get the model for nothing
What Makes a Record Replayable?
Three properties. Most production systems satisfy zero of them by accident, and all three by design once someone decides to.
This is why legibility is not an aesthetic preference. A readable record of decisions and their outcomes is replayable. An embedding of the same decisions is not, because "what happened next" was never a field in it. You can search a vector database beautifully and still be unable to ask it the one question that matters here.
What Can You Do With This Without a Research Budget?
More than you would think, because the entry cost is a logging decision, not a modeling one.
If your agent records what it tried, in what order, and what each attempt produced, you can already replay:
- a different stopping rule — would a lower step cap have cut any completed work?
- a different retry cap — did attempts 3 and 4 ever produce anything?
- a different batching strategy — which independent calls could have run together?
- a different escalation threshold — when should this have gone to a bigger model?
All at zero model cost, over runs you have already paid for.
The hard part is almost never the replay. It is that most systems store the decision in one place (a transcript) and the outcome in another (a billing log, a commit, a build status) with no key joining them. If you take one thing from this article for your own stack: make sure the record of what your agent decided carries an identifier that reaches the record of what happened next.
WHAT MOST SYSTEMS HAVE WHAT REPLAY NEEDS
────────────────────── ─────────────────
transcript ─── "step 7 ran X" transcript ─── "step 7 ran X" ──┐
│ join key
billing log ── "$0.31 at 14:02" billing log ── "$0.31, step 7" ─┤
build log ──── "failed at 14:02" build log ──── "failed, step 7" ┘ joined by WALL CLOCK joined by IDENTITY
(guesswork) (replayable)
The Honest Limits
Three, and the first is structural rather than fixable.
- A replay can only answer about territory you recorded. If a better strategy would have opened a branch nobody opened, there is no stored outcome and the replay simply cannot say. This is why the loop must alternate: replay offline to choose, go online to discover.
- The online half is stochastic and reported as single samples. The paper is explicit that the online transition is stochastic because the agent can produce different outcomes from the same starting workspace. Its offline replay is deterministic by construction and reproduces exactly; its online multipliers are one sample each. Before trusting any redeploy delta in your own system, establish your run-to-run variance first.
- A scalar objective hides trade-offs. The paper's replay score combines quality, an execution-cost penalty and a parallelism bonus into one number. That is appropriate for a benchmark with a single scoring protocol, and much less appropriate where quality is a human judgment across several axes that should not be averaged.
Why This Matters Beyond Research Labs
Because it reframes what an AI workspace is for.
Most products treat the record of work as exhaust — logs to be rotated, transcripts to be trimmed, a bill to be paid. That is as true of AI app builders as of agent frameworks. This line of research says the record is an asset that appreciates: every run makes the next strategy decision cheaper to get right, but only if the record is structured, outcome-carrying, and something a person can open.
That is the same conclusion we reached from the opposite direction, by measuring memory features that did not work. A record you can read is a record you can correct, and a record you can correct is a record you can replay. We wrote up that year of measurements in A Year of Agent Memory Experiments.
In Taskade, the record of how work was done lives in the same projects the work lives in — and you can browse what people have built that way in the app gallery — the same views, the same editing, the same sharing — so agents are grounded on something a person can open and argue with, and automations watch the object itself rather than a copy.
To be exact about what we are and are not claiming: Taskade does not ship a replay simulator or an automated strategy-tuning loop like the one this paper describes. What it gives you is the precondition — a structured record of your work that a person can open, correct and re-walk.
▲ ■ ● The record of what happened is cheaper than making it happen again.
Start a workspace that records how it was built →
Sources and Deeper Reading
Primary source
- Zheng, Wu, Zhang et al., Dream-RSI: Recursive Self-Improvement through Evolving Worlds, arXiv:2609.14858, 14 September 2026 — Google, Google DeepMind, University of Maryland, University of Virginia. Project site · Repository · Hacker News discussion
The lineage
- Sutton, Integrated Architectures for Learning, Planning, and Reacting (Dyna), ICML 1990
- Ha & Schmidhuber, World Models, arXiv:1803.10122, 2018
- Hafner et al., Dream to Control: Learning Behaviors by Latent Imagination, arXiv:1912.01603, 2019 · Mastering Diverse Domains through World Models, arXiv:2301.04104, 2023
- Novikov et al., AlphaEvolve: A Coding Agent for Scientific and Algorithmic Discovery, arXiv:2506.13131, 2025
- Lange et al., ShinkaEvolve, arXiv:2509.19349, 2025
- Zhang et al., Darwin Gödel Machine, arXiv:2505.22954, 2025
- Ye et al., Structured Scaling of AI Discovery Across Diverse Scientific Domains (the SimpleTES system), arXiv:2604.19341, 2026
- Ouyang et al., ReasoningBank, arXiv:2509.25140, 2025
- Zhang et al., Agentic Context Engineering, arXiv:2510.04618, 2025
The measurement series
- A Year of Agent Memory Experiments — four negative results and a measured noise floor
- Structure Beats Instruction — the same direction as this paper's §5.1, reached in a different setting
- Which AI App Builder Remembers Why? — which tools keep a replayable record at all
- Multi-Agent Systems in Production — what survives contact with real users
- The Execution Layer — why a record without execution is an archive
Concepts
- Recursive Self-Improvement · Replay Simulator · World Model
- Off-Policy Evaluation · Evaluation Noise Floor
- Agent Loop · Agent Harness · Exploration and Discovery
- Reinforcement Learning · Evals · Non-Determinism
- Agent Memory · Persistent Memory
Frequently Asked Questions
What is a replay simulator in AI?
A replay simulator is a completed run treated as an environment you can act inside. If a system records each decision it made alongside the outcome that decision actually produced, then an alternative strategy can be evaluated by walking that record in a different order and reading the stored results, instead of paying to run the whole expensive process again. The key property is that every outcome is already on disk, so evaluating a new strategy costs reading rather than execution.
What is Dream-RSI?
Dream-RSI is a framework published in September 2026 by researchers from Google, Google DeepMind, the University of Maryland and the University of Virginia, described in arXiv paper 2609.14858. It targets recursive self-improvement of the exploration strategy an AI coding agent uses during long automated discovery. It runs a three-stage loop: explore online and record a discovery tree, convert that tree into a replay simulator, then test many candidate strategies inside the simulator and redeploy only the winner.
Why is evaluating an AI agent exploration strategy so expensive?
Because the feedback is delayed and the search space is large. Judging one candidate solution is cheap, but judging a strategy means watching it steer an entire discovery run across many propose-and-evaluate cycles before you learn whether it was any good. The paper frames this as two simultaneous walls: meta-level feedback is delayed and expensive, and the space of possible strategies is vast, so most candidates you would need to try are bad ones that each cost a full rollout to rule out.
How much cheaper does Dream-RSI claim to make discovery?
On a Lasso regularization path task the paper reports reaching better downstream runtime using 317 discovery-agent calls versus 550 for its fixed-exploration control, and compares that to a prior system reported at 51,200 generations, which is where the up to 162 times figure comes from. On GPU kernel tasks it reports reaching comparable performance with 2.43 and 1.79 times fewer generations, or 2.09 and 1.44 times higher performance under comparable budgets.
Is a replay simulator the same as a world model?
They are related but not identical. A learned world model, as in the Dreamer line of work, approximates environment dynamics so an agent can imagine trajectories that never happened. A replay simulator does not approximate anything. It is an exact record of the portion of the space that was actually visited, so it cannot tell you about branches nobody explored. You trade coverage for exactness, and the record comes free as a by-product of working.
What is the main limitation of a replay simulator?
It can only tell you about the part of the search space you already recorded. If an alternative strategy would have opened a branch nobody ever opened, the replay has no outcome stored for it and simply cannot answer. This makes replay excellent for comparing how to order, batch and stop across known territory, and useless for discovering unknown territory. That is why the loop alternates: you replay offline to pick a better strategy, then go online to record new ground.
Why can the Dream-RSI loop not get worse?
Because the current strategy is included in the candidate set it is compared against. The paper selects the next strategy as the highest-scoring version among all evaluated versions, and since the incumbent is one of those versions, the selected strategy scores at least as well as the incumbent on the recorded history. That is a guarantee about the offline score on that fixed history, not a guarantee about future online performance, and it is a design pattern worth copying.
What makes a record replayable?
Three properties. It must be structured, so there are parent and child relationships rather than a flat log. It must carry outcomes, so each recorded decision sits next to the result it actually produced. And it must be re-walkable in a different order, so an alternative strategy can visit the same nodes in a sequence nobody used. A chat transcript usually fails the first, an embedding fails all three, and a tree of attempts with scores attached satisfies all three.
Did Dream-RSI find that injecting history into the prompt helps?
No, the opposite. The paper tested abstracting prior trajectories into high-level directional insights injected into the prompt as explicit semantic guidance, and reports that this consistently underperformed the unguided counterpart across both paradigms under equivalent budgets. The stated interpretation is that strong semantic bias about future search directions over-constrains the space and impedes diverse exploration.
How does this relate to Dyna and model-based reinforcement learning?
It is the same family of idea applied to a new layer. Richard Sutton's Dyna architecture in 1990 proposed learning from simulated experience alongside real experience. World Models by Ha and Schmidhuber in 2018 and the Dreamer line by Hafner and colleagues train policies inside a learned model of the environment. Dream-RSI applies the pattern at the meta level, where the thing being improved is the exploration strategy and the model of the environment is the literal recorded history.
Is Dream-RSI really recursive self-improvement?
Not in the sense most people mean. The model weights never change, the coding agent never changes, and the evaluator never changes. What gets rewritten between rounds is the code of the exploration strategy, a lightweight orchestration layer that decides which branch to continue, how many attempts to run in parallel and when to stop. When the paper was discussed on Hacker News this was the main objection raised, and it is a fair one. The useful idea is not self-improvement, it is that a completed run is a free simulator most teams currently discard.
What is the difference between Dream-RSI and AlphaEvolve?
AlphaEvolve evolves candidate solutions, meaning the programs being discovered. Dream-RSI evolves the strategy that decides how to search for those solutions, and it does so offline against recorded history rather than by paying for more live rollouts. They operate at different layers and are complementary rather than competing. AlphaEvolve asks what program to write next, while Dream-RSI asks how to spend the next hundred attempts.
Can a normal team use a replay simulator without a research budget?
Yes, and the entry cost is a logging decision rather than a modeling one. If your agent records what it tried, in what order, and what each attempt produced, you can already replay a different stopping rule, a different retry cap or a different batching strategy over that record at zero model cost. The hard part is almost never the replay. It is that most systems record the outcome and the decision in separate places with no key joining them.