Airbnb builds AI agents that review trust cases for guests and hosts against written policy, according to a conference talk by Pedro Rodriguez, Director of Engineering, ML at Airbnb, published on September 29, 2026.
TL;DR: Airbnb's trust agents make decisions where a wrong answer affects a real guest or host, and where someone can later ask why. Pedro Rodriguez describes five patterns (04:31): typed outputs with an abstain option that sends unsure cases to a human, small subagents behind a plain rules layer, agents that wait days for a signal, one controlled data layer for all agent reads, and checks on live runs. He reports that the first abstain design sent about 25% of cases to humans that the agent ought to approve (06:16). His main lesson: building the agents was easy, and trusting them took the most work. The talk gives no accuracy figures for the trust agents.
| Fact | Detail |
|---|---|
| Source | View source |
| Source type | YouTube talk (LangChain Interrupt NYC), 19 min |
| Source published | 2026-09-29 |
| Speakers | Pedro Rodriguez, Director of Engineering, ML, Airbnb |
| Industry | Travel and accommodation marketplace |
| Function | Trust and safety, customer support |
| Techniques | Structured output, abstain path, human review, rules layer, subagents, LLM judge, consistency checker |
| Tools named | LangGraph (interrupts), MCP |
Independent summary of public material. Airbnb is not affiliated with Taskade.
The system Airbnb built
Pedro Rodriguez leads the AI agents that protect the Airbnb community before, during and after a trip, for guests and for hosts. Some of these agents help a human with a case, and the human makes the final decision. Other agents decide cases on their own, and a human sometimes never sees the case.
Rodriguez defines high stakes in two ways. A wrong decision affects a real host or guest. And someone can come back later and ask why the agent made the decision, so the team must be able to explain it.
The talk opens with a failure. The team sent two copies of the same record to an agent and got two different answers that both looked right (00:16). Only a small consistency check in the workflow caught the problem.
Architecture of the system
Rodriguez describes five patterns (04:31):
| Pattern | What Airbnb does |
|---|---|
| Typed answers with abstain | The agent returns a typed object with a decision, a reason that names the part of the policy it used, and a summary. A malformed answer fails and goes to human review. The agent can also abstain, which sends the case to a human |
| Small subagents behind a rules layer | Several small subagents each handle part of a case, and a synthesizer combines their answers. A plain rules layer with no AI runs first. It sends a case to a human when policy requires it, and it stops early when data is missing |
| Agents that wait | Some cases wait days for an outside signal or a human answer. The graph pauses and resumes later. Case data lives in a separate database, so other systems can query across runs, for example to check service-level targets |
| One data layer | MCP is the only path from the agents to about 10 trust systems (10:49), with logging and access control. Agents make no direct API or database calls. Months later, the MCP logs answered a question about why an agent decided a case |
| Checks on live runs | The first agent got seven checks: five LLM judges and code checks (14:12). A separate checker on a different model family makes sure that two identical records get the same answer |
The abstain design needed tuning. Rodriguez reports that the first version abstained on about 25% of cases that it ought to approve (06:16). The team split the work into two calls (06:26). The first call finds the part of the policy that applies, and the second call decides with only that part. Cost went up and abstains went down.
The small-agent design also cost more. It added model calls, cost and latency. Rodriguez says that scheduling hides the latency, because nobody waits for these answers in real time, and the team accepted the extra cost.
Results the source reports
- Pedro Rodriguez reports that the first abstain design sent about 25% of cases to human review that the agent ought to approve (06:16), and that the two-call split reduced that rate (06:26).
- Rodriguez reports that the team runs code checks every day and samples automatic decisions for the LLM judges every week (14:43).
- For Airbnb's separate AI support assistant, Rodriguez reports that it resolves 45% of the contacts it handles with no human (03:19), and that it is live in more than 50 languages (03:00).
- Rodriguez says: "Building agents was the easy part, but knowing whether you can trust these agents is what is going to take you the most work." (18:00)
Critical assessment
The talk describes patterns, not outcomes, for the trust agents. It gives no accuracy, appeal or error rates for trust decisions. The 45% figure (03:19) belongs to the support assistant, a different system. The consistency checker has a limit that the speaker names himself: if two answers match and both are wrong, the checker does not catch it. The talk took place at a conference run by the vendor of the framework that the team chose. No outside party checked the claims. The five patterns (04:31) are specific and apply to any agent that makes decisions a person can appeal.
Build this in Taskade
- Build a team of AI agents with file analysis, persistent memory and multi-agent collaboration that checks each case against your written policy. Start at AI agents.
- Route each new case to the right person with support automations.
- Connect an outside MCP server through an automation with the MCP Client connector. The connector works on every plan.