Definition: Off-policy evaluation (OPE) estimates how well a strategy would have performed, using data that was collected while following a different strategy. It answers "what if we had done it this other way?" without running the other way.
TL;DR: Off-policy evaluation lets you score a change using runs you already paid for. It is what makes a replay simulator valuable, and its central weakness is coverage: it can only speak about situations the collected data actually visited. See it applied to agents.
Off-Policy Evaluation Explained in 3 Levels
Level 1, the everyday version. You kept receipts for a year. You can now work out what a different shopping habit would have cost you, without living that year again.
Level 2, the builder's version. Your agent logged every step and outcome last month. You want to know whether a lower retry cap would have hurt. You replay the logs under the new cap and count what changes — no model calls, no waiting.
Level 3, the architect's version. Given trajectories collected under a behavior policy, estimate the value of a target policy. Standard estimators include direct methods, importance sampling weighted by the ratio of target to behavior action probabilities, and doubly robust combinations. Variance grows sharply where the two policies disagree, and the estimate is undefined where the behavior policy had no support.
Why It Matters for AI Agents
Evaluating an agent strategy online is brutally expensive. Judging one output is cheap; judging a strategy means letting it steer a whole run, then changing one thing and paying again.
What You Can Evaluate Off-Policy
If your agent logs what it tried, in what order, and what each attempt produced:
| Question | Replayable? | Why |
|---|---|---|
| Would a lower step cap have cut completed work? | Yes | The recorded steps show where work finished |
| Did retries 3 and 4 ever produce anything? | Yes | Their outcomes are stored |
| Which calls could have run in parallel? | Yes | Dependencies are visible in the record |
| Would a different model have done better? | No | No stored outcome exists for it |
| Would a branch nobody opened have worked? | No | Unvisited territory has no data |
The last two rows are the boundary, and it is a hard one.
Where It Breaks
- No coverage, no answer. Where the collected data never went, no estimator can help. This is not a precision problem — the quantity is undefined.
- Variance explodes under divergence. The more the target strategy differs from the one that collected the data, the noisier the estimate. Estimates for radically different strategies are often worthless.
- Stale data. Logs collected under an older model or tool version describe a system that no longer exists.
- The join problem. In practice the most common blocker is not statistical. Most systems record the decision in one place and the outcome in another with no key linking them, so the trajectory cannot be reconstructed at all.
The Practical Requirement
Before any of this is possible, the record has to carry a join key — an identifier on the decision that reaches the record of what happened next.
WITHOUT A JOIN KEY WITH A JOIN KEY
────────────────── ───────────────
transcript: "step 7" transcript: "step 7" ──┐
billing: "$0.31, 14:02" billing: "step 7" ──┤
build log: "failed, 14:02" build log: "step 7" ──┘
joined by wall clock joined by identity
(guesswork) (replayable)
Related Concepts
- Replay Simulator · World Model · Recursive Self-Improvement
- Evaluation Noise Floor · Evals · Agent Evaluation
- Reinforcement Learning · Agent Observability · Non-Determinism
The prerequisite is a record with a join key. In Taskade, work and the record of it live in the same projects, which is what makes a decision traceable to its result.
Read next: Replay Simulators Explained — off-policy evaluation applied to a real agent system, with the reported results and the limits.