Every team building AI agents runs into the same wall. The demo works. The model writes the code, drafts the email, books the trip. Then it ships to real users, and one run in five does something wrong: invents a fact, skips a step, deletes the wrong row. The question everyone asks next is, "How do I make this reliable?"
The answer that keeps showing up, from a Swedish music company to a customer-service unicorn to a mathematics lab at DeepMind, is not a better model. It is a better system around the model.
TL;DR: You do not buy reliability from a model. You build it around one. Wrap an unreliable generator in a separate, fresh-context checker, and end at a deterministic gate that cannot lie. Spotify's LLM judge took its coding agent from roughly 25 percent to about 80 percent success. Build the loop in a workspace →
🧭 Why Do AI Agents Fail?
AI agents fail because a language model is non-deterministic: the same prompt can return different output on different runs, and a small early mistake compounds across a multi-step task until the whole result breaks. A single model is a brilliant guesser, not a guarantee. Reliability is a property of the system you build around it, and treating the model as if it were the system is the root cause of most agent failures.
Think about what a chained task actually asks of a model. If each step in a ten-step workflow is right 95 percent of the time, and the steps are independent, the whole chain finishes correctly only about 60 percent of the time (0.95 to the tenth power). Push each step to 99 percent and the chain still lands near 90 percent. The math is unforgiving: long agent runs amplify small error rates, which is why a model that looks flawless in a one-shot demo stumbles the moment it has to do twelve things in a row. See non-determinism for why "same input, same output" does not hold for language models.
The other failure source is over-confidence in the model's own judgment. Models exhibit sycophancy: they tend to agree, to rationalize, and to defend an answer they already committed to. Ask a model to grade its own work and it usually gives itself a pass. That single fact, that a self-check is a weak check, is the seed of the entire architecture below.
🏗️ The One Architecture Everyone Converges On
Across five separate fields, independent teams arrived at the same three-part shape: a generator that proposes, a separate checker that reviews with fresh eyes, and a deterministic gate that only passes correct output. Nobody coordinated this. Coding-agent teams, a customer-service company, and a formal-mathematics lab each rediscovered it under different names, which is the strongest signal that it is real.
The generator is the part you cannot fully trust, and that is fine. Its job is to produce candidates, not guarantees. The checker's job is to catch the bad candidates before they reach the world. The gate is the part that can never be wrong, because it is code, not a guess. Here is where the pattern shows up.
| Effort | Field | The generator | The checker or gate |
|---|---|---|---|
| Spotify Honk | Software engineering | Coding agent | LLM judge, then tests + CI |
| Vercel d0 | Data analytics | SQL agent | Query runs against real data |
| Manus | General agents | Task agent | Constrained action space + reviewer |
| Sierra | Customer service | Reasoning model | Supervisor model |
| DeepMind AlphaProof | Mathematics | Language model | Lean proof checker |
Five fields, and if you count the distinct efforts inside them, at least nine independent teams: three in coding alone, plus customer service, formal math, the LLM-as-a-judge evaluation literature, the fresh-context validator pattern, the tool-subtraction result, and the human-in-the-loop workspace approvals we will cover last. They disagree on almost everything else. They agree on the shape.
🧮 The Reliability Math: Why Two 90% Checks Beat One
Reliability compounds when you stack independent checks. Sierra founder Bret Taylor, the former co-CEO of Salesforce, put the math plainly on the Cheeky Pint podcast: "If you imagine a reasoning system is right 90 percent of the time but has some either guardrail, malfunction, or hallucination 10 percent of the time... Then you have a supervisor that's right 90 percent of the time. If you chain them together, you get 99 percent effectiveness."
The reason is simple probability. For a bad output to slip through, the generator has to make an error and the checker has to miss it. If each fails 10 percent of the time and they fail independently, the combined miss rate is 10 percent of 10 percent, which is 1 percent. Two mediocre 90 percent components produce a 99 percent system.
| Setup | Miss rate | Effective reliability |
|---|---|---|
| One model, right 90% of the time | 10% | 90% |
| Add a supervisor, also 90% | 10% × 10% = 1% | 99% |
| Add a deterministic gate on top | Near 0% on checkable output | Near 100% |
Sierra, a customer-service AI company valued at 4.5 billion dollars in 2024 per CNBC, runs what Taylor and co-founder Clay Bavor call a "constellation" of models: a primary model does the heavy lifting, and a supervisor watches the reasoning and sends the work back with notes when the agent drifts off script. That supervisor is a reflection pattern with teeth.
One caveat carries the whole idea: the checks must be genuinely independent. Two copies of the same model with the same prompt tend to make the same mistake, so their miss rates do not multiply, they overlap. Independence is what makes the second 90 percent worth anything. That is why the checker wants fresh context, a point we return to below.
🪜 The Verifier Ladder: LLM Judge to Deterministic Gate
The most valuable move a team can make is to treat the LLM judge as a bridge and the deterministic gate as the destination. Spotify's own sequence proves it. Its background coding agent, Honk, opens pull requests without a human in the loop, and the team reports that adding an LLM judge to validate output took pull-request success from roughly 25 percent to about 80 percent, per the Spotify x Anthropic agentic-development interview with VP of Engineering Niklas Gustavsson. Then the models improved, and Spotify removed the judge. Verification did not leave. It moved into tests, CI, builds, code review, and production safeguards.
That order is the citable lesson. The judge is scaffolding you use to reach a place; the deterministic gate is the place. Build the gate first when you can, and use the judge only for what the gate cannot yet cover.
In the same interview, verification was framed as "the single most important part of closed-loop agentic development," and both Spotify's Gustavsson and his Anthropic interviewer called it the place most companies underinvest. Spotify now reports that AI assists roughly 73 percent of its pull requests, a number that only holds up because the verification layer underneath it does. Here is how the two rungs of the ladder differ.
| Property | LLM judge (the bridge) | Deterministic gate (the destination) |
|---|---|---|
| What it is | Another model's opinion | Code: tests, schema, compiler, proof |
| Verdict | A score with reasons | Same answer every time |
| Can it be fooled? | Yes, by tone or confidence | No, it runs the check |
| Best for | Open-ended, hard-to-code output | Anything you can express as a rule |
| Cost over time | Recurring tokens per check | Write once, runs free |
If you write software, the deterministic gate you already own is a review pipeline and a test suite. See AI code review for how a separate reviewer fits the same generator-checker shape, and spec-driven development for turning fuzzy requirements into the kind of checkable rule a gate can enforce.
🔒 When the Checker Cannot Lie: AlphaProof and the Gold-Standard Gate
The purest version of this architecture lives in mathematics, where the gate literally cannot be wrong. In July 2024, DeepMind's AlphaProof, paired with its geometry counterpart AlphaGeometry 2, reached silver-medal standard at the International Mathematical Olympiad, together scoring 28 out of 42 points and solving four of six problems. AlphaProof itself cracked three of them, including the competition's hardest, which only five human contestants completed. The result was later published in Nature.
AlphaProof works by coupling a language model with the Lean theorem prover. The model generates candidate proof steps, which are exactly the kind of confident, sometimes-wrong output a language model produces. Lean then checks each step formally, and a proof that does not follow from the rules simply does not pass. As DeepMind puts it, formal languages offer "the critical advantage that proofs involving mathematical reasoning can be formally verified for correctness." The generator can hallucinate all it wants. The gate does not care about confidence; it only cares whether the logic holds.
This is the shape at its cleanest: an unreliable creative engine bolted to a verifier that never lies. You cannot always get a Lean-grade gate, because most real tasks are not formal proofs. But it sets the target. Whenever any part of an agent's output can be turned into a rule a machine checks the same way every time, that part should graduate off the LLM judge and onto the gate.
✂️ Fewer Tools, More Reliable: Vercel and Manus
Counterintuitively, one of the most effective reliability moves is subtraction: removing tools, not adding them. Vercel's engineering team built an internal text-to-SQL agent with a stack of specialized tools, heavy prompt engineering, and careful context management, and it topped out at 80 percent success. They deleted about 80 percent of those tools and gave the agent a single command-line tool to browse their data the way a human analyst would. Success rose to 100 percent, the agent ran 3.5 times faster (average time dropped from about 275 seconds to 77), it used 37 percent fewer tokens, and it took 42 percent fewer steps.
Vercel's own conclusion is the memorable part: they had been constraining the model's reasoning because they did not trust it to reason, and when they stopped making choices for it, the model made better ones. A smaller, cleaner action space gave the checker less surface area to police and the generator fewer ways to go wrong.
Manus, the team behind a widely used general agent, reached the same place from a different road. In their engineering write-up on context engineering, they report rebuilding their agent framework four times, a process they jokingly call "Stochastic Graduate Descent." A recurring lesson: piling on tools backfires, because "your heavily armed agent gets dumber." Rather than swapping tools in and out mid-task, which confuses the model and breaks prompt caching, they mask the ones that should not fire and keep the menu small and stable.
| Team | Before | After | Result |
|---|---|---|---|
| Vercel d0 | Many specialized tools | 1 command-line tool | 80% → 100%, 3.5x faster |
| Manus | Growing tool menu | Small, masked, stable set | Fewer errors, framework rebuilt 4x |
The through-line: a reliable agent is a focused agent. Every extra tool is another branch where the generator can pick wrong and another thing the checker has to understand. See agent harness for how the software around the model, not the model, is where this reliability actually lives, and context engineering for how what you put in the window shapes what comes out.
🥸 Why a Fresh-Context Checker Beats a Self-Check
A checker that starts with a clean slate is adversarial by design, and that is exactly what you want. When the same agent that wrote the answer is asked to grade it, it carries the reasoning that led there, so it defends its own work. A second agent with fresh context has no stake in the prior decisions. It reads the output cold and asks the honest question: does this actually hold up?
This is why teams run the checker as a separate subagent with its own context window rather than as one more turn in the same conversation. It is also why an agent handoff from a "doer" to a "reviewer" outperforms asking one agent to do both jobs in sequence. The reviewer's ignorance of how the sausage was made is the feature, not a bug. If you want the formal version of "grade the output, then decide," read agent evaluation and evals, which turn ad-hoc checking into a repeatable score you can track over time.
Fresh context also guards against a subtler failure. Long agent sessions suffer from context rot, where early instructions drift and errors accrete in the window. A checker that opens fresh does not inherit that rot. It sees only the artifact and the question, which is the cleanest possible position from which to judge.
🧩 How to Build This Loop in a Workspace
You do not need a research lab to run this architecture. The same three parts, generator, checker, gate, assemble inside a workspace, where the checker and the gate live right next to the work instead of in a separate pipeline. This is how Taskade is built: Workspace DNA is Memory plus Intelligence plus Execution, which maps one-to-one onto the loop.
Here is how each part maps to something you can set up today:
- The generator is a Taskade AI agent, or Taskade EVE, proposing the draft, the plan, or the change. It has a large built-in toolset and can pull from your projects, so it produces real candidates, not toy ones.
- The checker is an eval or a structured-output step that reads the agent's result and returns a pass, a fail, or a score. Structured output is your lightweight deterministic gate: if the result does not match the required shape, it fails automatically, no judgment call needed.
- The gate for the actions that matter is a human-in-the-loop approval. An automation can pause and wait for a person to approve before it sends the email, charges the card, or writes to the database of record. Approvals are the deterministic gate on consequences: nothing high-stakes ships until a human clicks yes.
- The execution is an automation action firing once the check passes, connected through 100+ bidirectional integrations so the approved result actually goes somewhere. Triggers pull events in; actions push the result out.
The point is not that any one feature is magic. The point is that reliability comes from the loop, and a workspace lets you place a checker and an approval gate between the model and the real-world action without wiring together a separate system. Browse the community gallery to see agents and automations other people have assembled into exactly this shape.
✅ The Reliability Decision Tree
Before you reach for a bigger model, work down this tree. It routes every piece of an agent's output to the cheapest check that actually catches its failures.
For each thing the agent produces, ask: Can I verify it with code (test, schema, compiler, rule)?
├─ YES ─► Build the DETERMINISTIC GATE. Ship only what passes.
│ (AlphaProof + Lean, Spotify's tests, Vercel's live query)
│
└─ NO ──► Is a wrong result high-stakes (money, data, safety, brand)?
├─ YES ─► HUMAN-IN-THE-LOOP approval before the action fires.
│ The person is the gate on consequences.
│
└─ NO / at scale ─► LLM JUDGE with FRESH context.
Use it as the bridge. As the task hardens
into a rule, graduate it to the gate above.
Then, across the whole system:
• Chain independent checks so miss rates multiply (90% x 90% = 99%).
• Cut tools you do not need. Fewer actions, fewer ways to fail.
• Keep the checker's context separate from the generator's.
Notice what is not on the tree: "use a smarter model." A better model raises the generator's hit rate, which helps, but it does not change the shape. Spotify removed its judge only after the models improved, and even then it kept the deterministic gate. The model is the engine. The verification loop is the brakes, the mirrors, and the seatbelt, and you would not ship a car with only an engine.
📚 Related Reading
- History of the Agent Harness: The Software Around the Model: how the industry learned that the harness, not the model, carries reliability.
- History of AI Benchmarks: Why Every Model Claims to Be Best: why leaderboard scores rarely predict real-world reliability.
- History of AI Agents: From SHRDLU to the Agent Loop: the long road to the perceive-plan-act-check cycle.
- How LLMs Got Hands: The History of Tool Use: where the "tools" in "fewer tools, more reliable" came from.
Frequently Asked Questions
Why do AI agents fail?
AI agents fail because a language model is non-deterministic: the same prompt can produce different output on different runs, and a small error early in a multi-step task compounds into a broken result. Reliability is a property of the system you build around the model, not of the model alone. The fix is to wrap the generator in a separate checker and a deterministic gate that catches bad output before it ships.
How do you make an AI agent reliable?
Wrap the model in verification. Keep a generator that proposes the work, add a separate checker with fresh context that judges the output, and end at a deterministic gate (tests, a schema, a formal rule) that can only pass correct results. Spotify reports that adding an LLM judge took its coding agent's pull-request success from roughly 25 percent to about 80 percent. Build the deterministic gate first and use the judge as the bridge to reach it.
What is LLM as a judge?
LLM as a judge is a pattern where one language model evaluates the output of another and returns a pass, a fail, or a score with reasons. It works well for open-ended output that code cannot easily check, such as tone, completeness, or whether a summary is faithful. It is a bridge, not a destination: as a task hardens into a checkable rule, replace the judge with a deterministic gate that cannot be fooled.
What is the difference between an LLM judge and a deterministic gate?
An LLM judge is another model giving an opinion, so it is fast to build and flexible but can be wrong or gamed. A deterministic gate is code: a test suite, a schema validator, a compiler, or a formal proof checker that returns the same verdict every time and cannot hallucinate. The judge handles what you cannot yet check with code. The gate is where you want every agent to end up.
Should you use an LLM judge or deterministic checks?
Use deterministic checks whenever the output can be verified by code, because they are exact and cannot be fooled. Use an LLM judge for output that resists a hard rule, or as a stepping stone while you build the gate. Spotify sequenced it exactly this way: it added an LLM judge first, then removed the judge as models improved and let deterministic tests, CI, and builds carry verification.
Does adding more tools make an AI agent more reliable?
Usually the opposite. Vercel's engineering team removed about 80 percent of a data agent's specialized tools, gave it one command-line tool, and success rose from 80 percent to 100 percent while the agent ran 3.5 times faster and used 37 percent fewer tokens. Manus, which rebuilt its framework four times, warns that a heavily armed agent gets dumber. Fewer, well-chosen actions beat a large menu of narrow ones. See context engineering.
What is a fresh-context checker or validator?
A fresh-context checker is a second agent that reviews the first agent's output without seeing the reasoning that produced it. Because it starts clean, it has no stake in defending prior decisions, so it evaluates adversarially rather than rubber-stamping. This is why a separate subagent catches errors a self-check misses: the self-check tends to agree with itself, a form of sycophancy.
How reliable can chaining two 90 percent checkers be?
Sierra's Bret Taylor describes the math: a reasoning system that is right 90 percent of the time chained with a supervisor that is right 90 percent of the time reaches about 99 percent effectiveness, because both have to miss the same error for it to slip through. Layers of independent checks multiply reliability. This only holds when the checks are genuinely independent, not two copies of the same model repeating the same mistake.
Did Spotify keep its LLM judge?
No. Spotify added an LLM judge to its background coding agent, which took pull-request success to about 80 percent, then removed the judge as the underlying models improved. Verification did not become less important. It moved into standard engineering infrastructure: tests, CI, builds, code review, and production safeguards. The deterministic gate outlived the judge.
How do you build a reliability loop in Taskade?
In Taskade you assemble the same loop inside a workspace. An AI agent proposes the work, a structured-output check or eval judges it, a human-in-the-loop approval step gates high-stakes actions, and automations run the approved action, all recorded back to the project. Projects hold the memory, agents supply the intelligence, and automations handle execution, so the checker and the gate live next to the work.
Can you make an AI agent 100 percent reliable?
In narrow, checkable domains, yes: DeepMind's AlphaProof pairs a language model with the Lean proof checker, which formally verifies each proof, so a wrong answer cannot pass. Vercel reached 100 percent on an internal data task after simplifying it. For open-ended work, aim for compounding reliability through layered checks and a human gate on the actions that matter, not a single perfect model.
Reliability is not a model you buy. It is a loop you build: a generator that proposes, a fresh-context checker that reviews, and a deterministic gate that only passes what is actually correct. Nine teams across five fields, from Spotify's tests to Sierra's supervisor to AlphaProof's Lean verifier, converge on that shape because it is the shape that works. Put the checker and the gate next to your work, and the model's brilliance finally becomes something you can depend on. Build the loop in your workspace →
▲ ■ ●