Definition: The evaluation noise floor is how much a measured result moves when you change nothing. You run the same input, on the same model, through the same harness, more than once, and record the spread. Any reported improvement smaller than that spread is not evidence of an improvement.
TL;DR: Rerun your evaluation with nothing changed and measure the gap. In one set of paired runs, the gap between two identical runs ranged from 10.5% to 50.0% across task types, with a mean of 29.6%, while aggregate totals moved only 5.2%. Two runs give one observed gap, not a distribution, but a gap that size already dwarfs many single-run improvements reported in agent work. See how it was measured.
Evaluation Noise Floor Explained in 3 Levels
Level 1, the everyday version. Step on a bathroom scale twice in a row. If it reads two different numbers, you cannot trust it to tell you about a half-pound change.
Level 2, the builder's version. You changed a prompt and your benchmark improved 20%. Before celebrating, run the unchanged version twice. If those two runs already differ by 25%, your 20% told you nothing. More reruns then tell you how wide the spread really is.
Level 3, the architect's version. Sampling temperature, tool-call ordering, harness scheduling and provider-side routing all inject variance into a nominally fixed pipeline. Without a characterized variance distribution per measured quantity, a single-sample A/B has no power, and a selection loop built on it optimizes toward noise.
Why It Matters
Because the field reports single-run deltas constantly, and the noise floor is frequently larger than the delta.
Measured Examples
Independent groups measuring different things keep landing in similar territory.
| Source | What was varied | What moved |
|---|---|---|
| Taskade paired runs | Nothing at all | Gap between two identical runs: 10.5%–50.0% per task type, mean 29.6%; aggregate totals 5.2% |
| METR, Time Horizon 1.1 | Only the evaluation harness | GPT-4o measured time horizon: 9.2 min → 6.0 min |
| Chroma, Context Rot | Only haystack structure | Coherent text scored worse than shuffled, across all 18 models tested |
The pattern across all three: the thing you are measuring moves substantially when you are not touching it.
Magnitudes vs Categorical Outcomes
The noise floor does not apply equally to everything you measure.
| Outcome type | Example | Needs repeated runs? |
|---|---|---|
| Magnitude | Steps taken, tokens used, percentage improvement | Yes — each run is one sample from a distribution |
| Single-artifact fact | Did this build produce a working artifact? | No — it is a fact about that one artifact |
| Rate across runs | How often does the rule fire? How often does a build succeed? | Yes — a count of successes is a sample too |
This distinction keeps noise-floor discipline from paralyzing measurement without letting counts off the hook. "The mandated question fired on 0 of 4 arms" is an observed count, and it is not a zero rate. With four trials, a true firing rate above one half is still consistent with seeing zero (the 95% upper bound is about 60%). "Steps dropped 12%" is a magnitude, and it needs repeated runs before it means anything.
How to Establish Your Own
- Pick your measured quantities. Not one aggregate — the individual measurements you actually report.
- Run the unchanged pipeline more than once. Same input, same model, same harness, no code change. Two runs give one observed gap, which is a warning sign. Five or more start to show the actual spread.
- Record the spread per quantity, not just overall. Aggregates are far tighter than their components, and reporting only the aggregate hides the real uncertainty.
- Set an attribution threshold. One conservative working rule: do not attribute a single-run, per-arm change smaller than the largest gap you observed between identical runs.
- Re-measure after any harness change. The floor is a property of the whole pipeline, not of the model.
Common Mistakes
- Reporting the aggregate gap as the floor. Totals are tighter than their parts. Using the total's stability to justify a per-arm claim understates uncertainty, often by several times.
- Assuming temperature zero removes it. Tool ordering, scheduling and provider-side routing all remain sources of variance.
- Treating the model's floor as the system's floor. METR's result shows the harness alone can move a headline number.
- Treating two runs as a distribution. One pair of runs shows that noise exists. It does not tell you its size with any confidence.
- Reading a zero count as a zero rate. 0 of 4 is an observed count with wide uncertainty.
- Running a conditionally firing treatment without measuring its firing rate. If a treatment only fires sometimes, a small A/B can produce zero treated runs and a result indistinguishable from a clean null.
Related Concepts
- Evals · Non-Determinism · Temperature
- Agent Evaluation · LLM as a Judge · Agent Observability
- Off-Policy Evaluation · Replay Simulator · Context Rot
Read next: A Year of Agent Memory Experiments — where the 29.6% figure came from, and the four negative results it puts in context.