A Year of AI Agent Memory Experiments: Four Negative Results (2026)

In the Black Mirror episode "Eulogy," a man named Phillip is asked to contribute memories of a woman he once loved. He cannot. Years earlier he cut her face out of every photograph he owned, and what survives is a set of images with a hole where a person used to be.

A guide walks him back into the pictures. He recovers all of it — the birthday call, the proposal in London, the letter he never opened, her face. The reconstruction is total.

Then he attends the funeral, and she nods at him from across the room, and that is the entire yield.

Perfect memory, arrived too late to act on.

TL;DR: We spent a year measuring AI agent memory. A proactive recall feature returned null on 31 of 31 model calls, and an offline replay over 346 stored runs showed why the experiment could never have detected anything. Two identical runs, with nothing changed, differed by 29.6% on average across task types. Two things worked, and neither was a prompt. Build an agent that remembers →

What Failure Mode Are We Actually Naming?

The failure mode is not forgetting. It is remembering perfectly with no path to execution.

That distinction is the whole argument of the Memory Reanimation Protocol essay, and in September 2025 it fit in one line: memory reanimated must mean execution. A system that reconstructs the past beautifully and cannot act on it has not solved the memory problem. It has built a very good archive of a problem it is still having.

This started as a metaphor, and metaphors are cheap. What follows is the part where it met measurement — four results that did not go our way, one noise check that puts them in context, and the two changes that survived.

Why Do AI-Generated Apps Become Unmaintainable?

An AI-generated app becomes unmaintainable because the reasoning behind its shape is discarded at the moment the build completes. The code survives. The decisions do not. Six weeks later someone needs one change, and the file tree offers no answer to the only question that matters: why is it like this?

This is the industrial version of Phillip's outcome. The generation ran. The artifact exists. Nobody can act on it.

Generative builders produce a specific kind of dead shell: a rendered interface over a data model that was never created, an agent with nothing underneath it, an automation that never fires. It looks like an app the way a photograph of a house looks like a house. You find out which parts were never real only when you try to use them.

The standard industry answer is to write a longer specification. Front-load more instruction. Add more rules to the system prompt. The next four sections are why that answer is wrong, measured rather than argued.

What Exactly Evaporates When a Build Finishes?

Four kinds of knowledge die at the same instant, and none of them is the code.

A human engineering team writes all four down as a matter of routine — tickets, commit messages, a design doc, a thread someone can search. A generative build writes down none of them. You can see the consequence at scale in the public app gallery: thousands of working apps, almost none of which carry the reasoning that produced them. That is not a model failure. The model did exactly what it was asked. The loss is structural, and it is the same loss whether the builder is Taskade Genesis, a code generator, or a person working alone at 2am.

What Did We Try First, and Why Did It Not Ship?

We wrote it down. We wrote a great deal of it down: long design documents for a full long-term memory architecture, with a vector store and a knowledge graph.

None of them shipped as written.

What actually shipped was a small fraction of that design: a short instruction and an agreed place to keep the record.

That asymmetry is the first suspicion in this whole story, and it took another eight months to state properly: the thing we wrote was not the thing that changed behavior. A large document describing a memory system is not a memory system. It is a description, and descriptions do not change the starting state of anything.

What Happens When a Memory Feature Never Fires?

It returns null 31 times and produces an experiment with no treatment in it.

We shipped proactive recall — the obvious feature, the one nearly everyone building in this space is building. Whenever older context gets dropped from a long conversation, find the relevant memory and inject it back into the model. Straightforward, defensible, and exactly what an architecture diagram wants.

The recall function returned null on 31 of 31 model calls. Every one. The treated arm of the experiment was byte-identical to its control, because the code path under test never executed. We deleted the module.

That is the embarrassing version. Here is the part that makes it publishable.

Rather than rerun the rounds, we took the real shipped function and replayed it offline across all 346 stored run records we had, at zero model cost, and read off how often it would have fired: 187 of 346, or 54 percent.

Which retroactively invalidated the experiment's own design. A treatment that fires on roughly half of eligible runs is a coin flip, and a three-versus-three A/B on a coin flip has a genuinely good chance of landing zero treated runs — which is precisely what a clean null result looks like from the outside.

The standing rule this produced, which costs nothing to adopt and is the most portable thing in this article:

Any A/B test on a conditionally firing treatment must have its firing rate measured before its sample size is chosen.

Note what the rescue actually was. We did not rerun anything. We read a record of what had already happened, in a different order, and learned something the live experiment could not tell us. Hold that — it comes back in the last section, and it turns out to have a name.

How Much Does an LLM Benchmark Move When Nothing Changes?

More than most published improvements. We ran the same prompt, on the same model, through the same setup, twice, and measured the gap for each task type.

That last row is the one that makes the rest credible rather than alarming. Aggregates held steady. Individual measurements did not. If you report a per-arm number from one run, you are reporting a sample from a distribution you have not characterized.

Two runs do not characterize it either. They give one observed gap per task type, which is enough to show the noise is large and not enough to size it precisely. A proper band needs more repeats than we ran.

The consequence, stated once and without any pleasure: most single-run improvements reported in this area sit inside that band.

The working rule we adopted: because the largest gap we saw between two identical runs was 50 percent, we stopped attributing any single-run, per-arm change smaller than that. It is a conservative rule drawn from limited data, not a statistical threshold.

Counts need the same care. Whether one specific build produced a working artifact is a fact about that build. But a count across runs, such as 0 of 4, is an observed count with wide uncertainty, not a zero rate. With four trials, a true rate of more than one half is still consistent with seeing zero.

This is the most generous thing in this article, and we are aware of that. Anyone can take the 29.6 and apply it to our numbers too. Please do. A measuring stick that only measures other people is not a measuring stick.

We Are Not the Only Ones Finding This

Our number comes from one setup and two runs, so it is worth knowing that independent groups measuring completely different things keep landing in the same territory.

METR held the model and the task set constant and changed only the evaluation infrastructure — same model, same tasks, different harness. GPT-4o's measured time horizon went from 9.2 minutes to 6.0 minutes on infrastructure choice alone. Nothing about the model changed.

METR is careful about what that does and does not establish, and the distinction is the whole lesson: the shift that reached statistical significance was in task success rates for two of five models, while the time-horizon estimates themselves still fell inside the earlier bootstrapped confidence intervals. Which is the point. The headline number moved by a third, and the measurement was too noisy to call that move real.

Chroma's context rot research — 18 models, 194,480 LLM calls — found something stranger and more useful: a coherent, well-organized document is harder for a model to search than the same content shuffled into random order. Consistently, across every model they tested. Which quietly invalidates the advice, repeated in almost every context-engineering guide including some of ours, to always organize your context well.

Three groups, three methods, one conclusion: the thing you are measuring moves a lot when you are not touching it. If your evaluation has never been rerun unchanged, you do not have an evaluation — you have an anecdote with a decimal point.

What Actually Worked?

Two things, and neither was a prompt.

The first is that a change to the agent's environment beat a rule written into its instructions in every round where we tested both. One example, because it is the cleanest: we changed what a single tool said when the agent guessed a wrong file path. Nothing else. Tool errors fell from 7 to 3 and steps from 28 to 17. That is one run per arm, so treat the size of the drop with the caution the section above asks for. The direction is the finding.

Against that, a mandated clarifying question — written explicitly into the instructions — fired on 0 of 4 arms across three model families, all four of which hit exactly the ambiguity it was written for. Four arms cannot prove the rule never fires. They do show it failed every time we gave it the chance.

The second is the offline replay that rescued the recall experiment: reading a stored record of past work, in a new order, taught us more than another live run could have. The last section comes back to it.

The agent learns from the environment's response, not from being told in advance.

This finding is large enough, and useful enough outside our own stack, that it gets its own article: Structure Beats Instruction covers the cases in full, the economics of why a prompt rule is expensive and an artifact is cheap, and the independent replication of the same direction in a September 2026 paper from Google and Google DeepMind.

What Does Memory Look Like If Structure Wins?

If structure beats instruction, then memory should be structure rather than instruction. That single move decides the architecture.

Two roots, holding two different substances:

The record is not a log file and not a sidecar service. It is an ordinary Taskade project — same views, same editing, same sharing. You can open it, read it, and argue with it. That last verb is the entire design.

There is also an order to how the layers get built:

     storage  ─────►  intelligence  ─────►  execution  ─────►  interface
   (projects)        (agents grounded      (flows wired      (the app
                      on that storage)      to both)          people see)
 ▲ build order runs left to right
 ▼ failure mode runs right to left: an interface built first
   is a photograph of the app underneath it

Build the interface first and you get a convincing screen with nothing underneath it. Build the storage first and every later layer has something real to stand on.

Why Does One Record Need Four Consumers?

Because the alternative is four copies that disagree with each other by Thursday.

A single project row is simultaneously read by four different kinds of consumer, without a synchronization layer between them:

There is no sync layer because there is nothing to sync. The agents and the automations are not integrated with the data — they are pointed at it.

Why Is Legibility the Whole Bet?

Because a person can correct legible memory and cannot correct opaque memory.

This is the Bill Atkinson argument, and it is the strongest claim in the architecture. HyperCard's real contribution was never the scripting language. It was that the data model was the thing on the screen. A person who was not a programmer could open a stack, see the cards, change a field, and understand exactly what had changed.

A vector index cannot be corrected by the person it is wrong about. A markdown record can — and it is the only kind of memory you can replay.

Both halves of that sentence carry weight, and the second half is newer than the first.

The humane half: if an agent believes something wrong about a customer, and that belief lives in an embedding inside a vector database, the customer has no route to the mistake. There is no row to edit. If the same belief lives in a readable record of decisions, anyone can open it and fix the line.

The engineering half: a record that is structured, carries outcomes, and can be walked in a different order is replayable. That is what let us score the recall function across 346 run records without paying to rerun a single one. An embedding is not replayable, because "what happened next" was never a field in it.

That last row is real and we are not pretending otherwise. Vector search is excellent at the job it has. It is simply not a memory a human being can argue with, and arguing with it is the requirement.

The discipline that follows: memory is a prior, not an authority. If the record disagrees with what a tool just returned, the tool is right. Fix the entry and move on. A memory system that outranks live evidence is not a memory system, it is a hallucination with a filing cabinet.

What Can We Still Not Prove?

Four things. Stating them is why the rest of this article is believable.

1. That the build order reduces dead shells. We observed the defect by hand and observed the fix by hand. We never ran a controlled comparison.
2. That describing a task as a story beats describing it as a checklist for the same underlying steps. No ablation exists. It is a hypothesis with a good story attached, which is the most dangerous kind.
3. That edit N gets cheaper because edits 1 through N-1 were recorded. This is the compounding claim — the one that would justify the entire architecture commercially — and it is unmeasured. We now know how we would measure it: replay the recorded work and score the alternative offline. We have not built the loop that does it.
4. That a legible store beats an opaque one on task outcomes. Argued above, and argued well, but never measured against a vector baseline.

We could cut this section and the article would read stronger. It would also be worth less, because the four negative results above are only credible in the company of the four open questions here.

The Carol Test: One Prompt You Can Run Anywhere

Here is a single acceptance test you can run against any AI builder, including ours, including a competitor's.

Type this:

Help me remember Carol

Pass requires four layers and at least one path that runs end to end:

Fail is a beautiful memorial page with nothing behind it. A photograph of an app. Phillip's outcome.

Run it against whatever you use. We would rather you checked than believed us. Run it here.

What Compounds From Here

The last section left one thread loose on purpose. When recall came back null, we rescued the experiment by reading a record of work that had already happened, in an order nobody had walked before, at zero cost.

In September 2026 a team at Google and Google DeepMind published that move as a method. Their framing is that a completed run is not merely history — it is a simulator, because every decision in it is stored next to the outcome that decision actually produced. Score an alternative strategy by reading, deploy only the winner. We wrote a plain-English walkthrough of it: Replay Simulators Explained.

We have done that replay four or five times. Every one by hand, every one to settle a single argument. We have not built the loop, and this article is not a claim that we have.

What we have built is the precondition: a record that is structured, legible, and owned by the person it describes. A workspace that remembers how it was built is a workspace that can eventually get better at building. That is a direction, not a roadmap, and it has no date attached to it.

I built the first version of Taskade in a flooded basement on a hand-me-down machine, with nobody around to ask for permission. The point was never that the machine remembers. The point is that you can read what it remembered, and fix it.

▲ ■ ●  Memory, Intelligence, Execution — in that order, because the surface needs something underneath it.

Build something that remembers being built →

Deeper Reading

The architecture and the lineage

• Memory Reanimation Protocol — the architectural piece this article is the measured sequel to
• Structure Beats Instruction — the positive finding in full, with the independent replication
• Replay Simulators Explained — the method behind the 346-record rescue
• The Execution Layer — why memory without execution is an archive
• The Genesis Equation — memory as a multiplier rather than a feature
• The Living App Movement — what a system that keeps running looks like
• Which AI App Builder Remembers Why? — 10 builders checked for the record this article argues for

Evaluation and context

• Context Engineering: The Complete Field Guide — the five layers, and the anti-patterns
• A History of Prompt Engineering — how the field arrived at context
• The Frontend Playground Era — why generated interfaces outrun their foundations
• Multi-Agent Systems in Production — what actually survives contact with users

Concepts

• Agent Memory · Memory Types · Persistent Memory
• Context Window · Context Rot · Context Compaction
• Evals · Non-Determinism · LLM as a Judge
• Agent Evaluation · Agent Harness · Agent Loop
• Retrieval-Augmented Generation · Knowledge Graph

Frequently Asked Questions