Agent-Ready Website

Definition: An agent-ready website is a site that an AI agent can read, navigate, and act on as easily as a person can. It serves clean Markdown copies of its pages, publishes an llms.txt index, offers a Model Context Protocol (MCP) server for actions, sizes its rate limits for agent loops, and returns the same facts on every surface.

TL;DR: A website built only for people makes an agent parse layout, guess at prices, and retry blind. An agent-ready website passes five checks: Markdown copies of each page, an llms.txt index, an MCP server, rate limits sized for agent loops, and consistent data. Taskade's own site serves Markdown copies of every /docs, /learn and /blog article and a hosted MCP server on every paid plan. Build an agent that reads the web →

Think of a restaurant with two doors. The front door has the dining room, the lighting, and the printed menu: that is your HTML, made for people. The kitchen door is where the suppliers come in. They do not need the candles. They need a clear list of what you take, when you take it, and who signs for the delivery. An agent-ready website builds the kitchen door on purpose, so the agents that now arrive every day do not have to climb in through the dining room window.

Why Agent-Ready Websites Matter in 2026

In 2026, the people who build the web's infrastructure began to describe AI agents as a customer in their own right. Vercel closed its Ship 2026 conference in San Francisco with a fireside between Guillermo Rauch and Shopify CEO Tobi Lütke titled "The Third Customer", about software agents that shop on people's behalf (Vercel Ship 2026 recap, October 15, 2026). In the same recap, Notion CEO Ivan Zhao put it plainly at Ship NYC: "your next customer isn't only a human, it's a human and an agent, and agents are blind."

Two weeks later, Shopify's Fall '26 Edition for developers stated that "every Shopify store is agent-ready by default" (Shopify, October 28, 2026). When the largest commerce platform makes agent readiness a default, every other site gets measured against it.

Rauch gave the clearest test for it in a Delphi interview published October 29, 2026. In his words, "MCP is almost like the new HTTP." HTTP delivered HTML to people. The new questions, as he framed them, are whether you give agents digestible Markdown when they browse your site, and MCP so they can transact faster. He went further: a tool that refuses to expose its data and capabilities to agents breaks the loop, and he would delete such tools "like the plague." That is his opinion, not an industry rule, but it is the buying pressure an agent-hostile site now faces.

The top half is the web as it was built. The bottom half is the path an agent takes on a site that expects it: an index tells the agent what exists, a Markdown copy gives it the text with no layout to parse, and an MCP server lets it act. Both halves serve the same facts. Only the door differs.

The Five Parts of an Agent-Ready Website

Guides to agent readiness list many files and headers. Five of them do the real work, and the last two are the ones most sites miss, because they are about behavior, not files.

1. Markdown copies of each page

A Markdown copy is the same page with the navigation, scripts, and styling stripped out. The llms.txt proposal suggests serving it at the same URL with .md added, so /blog/taskade-mcp-server becomes /blog/taskade-mcp-server.md. An agent then reads headings, lists, tables, and links directly, and spends its context window on content instead of markup.

Two details separate a good copy from a liability:

• A canonical link back to the HTML page. Send a Link header with rel="canonical" that points at the human page. Search engines then keep ranking the HTML page, and the copy never competes with it as duplicate content.
• A separate URL, not a different answer on the same URL. Some guides suggest content negotiation: return Markdown when a request sends Accept: text/markdown. Many CDN caches key a response on the URL and ignore the Accept header, so a cached Markdown reply can be served to a person, or a cached HTML reply to an agent. Negotiation is safe only when every cache in front of the site varies on Accept. A distinct .md URL caches cleanly with no extra setup.

Here is what a well-formed copy returns on taskade.com, checked on November 7, 2026:

$ curl -sI https://www.taskade.com/blog/taskade-mcp-server.md
content-type: text/markdown; charset=utf-8
cache-control: public, s-maxage=3600, stale-while-revalidate=86400
link: <https://www.taskade.com/blog/taskade-mcp-server>; rel="canonical"

2. An llms.txt index

llms.txt is a Markdown file at the site root, proposed by Jeremy Howard on September 3, 2024. Only one part is required: an H1 with the name of the site. By convention it adds a short blockquote summary, then H2 sections that list links with a one-line note each, and an "Optional" section that an agent can skip when it is short on context. A large site can also publish an llms.txt under a subpath that covers only that section.

The file is a reading list, not an interface. It creates no tools, no live data, and no permissions. Its job is to send the agent to the right Markdown copy on the first try instead of a crawl. Pair it with a sensible robots.txt, which decides which crawlers may read the site at all. For tools that write both files, see the best AI robots.txt and llms.txt generators.

3. An MCP server for actions

Markdown and llms.txt let an agent read. An MCP server lets it act: search a catalog, create a record, run a job. Discovery happens at runtime, because an MCP client asks the server which tools it offers, then calls them through tool calling. An MCP server almost always wraps an ordinary API underneath. MCP vs API covers that split in detail.

An agent-ready MCP server asks for credentials and refuses a request without them. Taskade's hosted endpoint at https://www.taskade.com/mcp answers an MCP client that sends no credentials with 401. A person who opens the same URL in a browser gets a setup page instead. A server that does whatever any caller asks is not agent-ready. It is an open door, and prompt injection and tool poisoning make that door expensive.

4. Rate limits sized for agent loops

Rauch's warning in the Delphi interview was about scale: agents call your APIs "at a million times the rate that a human would," and clunky rate limits break their loops. His number is a figure of speech, not a measurement. The point stands: an agent loop retries, paginates, and fans out, so a limit tuned for a person clicking a button fails the first real agent.

Sizing for agents does not mean no limits. It means limits an agent can read and plan around:

• Return HTTP 429 with headers that state the budget, the requests left, and the seconds until the window reopens.
• Key the budget to the account, so a second token does not double the load, and give heavy endpoints such as bulk export their own, tighter budget.
• Publish the rules, so an agent's harness can back off on schedule instead of hammering a closed door.

Taskade's REST API guide follows this shape: a rate-limited response carries the budget, the requests left, and the reset time in headers, the authenticated budget is keyed on the account rather than the token, and bundle export and import run on a tighter budget than ordinary reads and writes.

5. Consistent data: no "APIs that lie"

The fifth part has no file and no header. Rauch named it in the same interview: much of modern software stands on APIs "that lie to you or break down all the time," and the brittle workarounds that people built around them now break "under agentic pressure." A person notices that a price looks wrong. An agent copies the wrong price into an order.

Consistent data means:

• One fact, one answer. The HTML page, the Markdown copy, the llms.txt summary, and the API return the same price, limit, and status.
• Real errors. An unknown page returns 404, not a 200 with an empty body, the "soft 404" that convinces an agent a page exists.
• Structured outputs where it counts. A tool returns typed fields, as with structured outputs, so the agent does not parse a sentence to find a number.
• A check before release. A test that fails the build when two surfaces disagree catches the error before an agent quotes it.

Agent-Ready vs Agent-Hostile: The Signs

Most sites are neither fully ready nor fully hostile. Use this table to place yours.

Run these five checks in order. The first "no" is the next fix.

A Worked Example: How taskade.com Speaks Agent

Taskade's own marketing site is a working example of the five parts. Each row below was checked live on November 7, 2026.

Two automated checks keep these surfaces honest, which is part 5 in practice. On every pull request, yarn qa:agent-markdown checks the Markdown routes, the .md URL rewrites, the cache and canonical headers, and the Markdown converter. yarn qa:agent-discovery checks the agent entry points: the Link header that points to /llms.txt, /SKILL.md, and the MCP server card, the agent rows in the site footer, and the single "For agents" menu module. The Markdown URL of an unknown or unpublished article returns a real 404, and each copy is built from the same source file as its HTML page, so the two stay in step.

The scope is honest, too. The Markdown copies cover the three article sections today: /docs, /learn, and /blog. The read path is public. The action path through MCP needs a paid plan and the user's own credentials, which is how it should be.

How to Make Your Website Agent-Ready

Work through these steps in order. Each one is useful on its own, so you can stop after any step and still be better off.

1. List the pages an agent needs. Usually docs, pricing, help articles, and product pages. Skip the rest at first.
2. Serve a Markdown copy of each one at the page URL plus .md, with a canonical Link header to the HTML page and a cache header of its own.
3. Publish /llms.txt. One H1, a one-paragraph summary, and H2 lists of links to the Markdown copies.
4. Offer an action path. Start with a documented API. Add an authenticated MCP server when agents need to search, create, or run things.
5. Make rate limits readable. Return 429 with budget and reset headers, key budgets to the account, and publish the rules.
6. Fix your errors. Unknown URLs return 404. Server faults return 5xx and are not cached as success.
7. Add a consistency check. Fail the build when the price, limit, or status on one surface differs from another.

Rauch's advice on the order of work fits here: test from the boundary, close to what the user sees, rather than with many small unit tests. For an agent-ready site, the boundary test is simple. Point a real agent at the site, give it a task, and read where it gets stuck.

How Agent-Ready Websites Connect to Taskade

Taskade sits on both sides of the kitchen door. As a destination, your workspace is reachable by outside agents through the hosted MCP server on every paid plan, so Claude Desktop, Cursor, or any MCP client can read and update your projects and agents. As a visitor, a Taskade AI agent uses built-in tools to search and read the web, and an automation's MCP Client step calls tools on another site's MCP server, on every plan.

When an agent is about to do something that matters, such as send an email, post a message, or spend money, put a person in the loop. The Wait for approval step pauses an automation until a workspace member approves or rejects. That is the human-in-the-loop pattern, applied where the cost of a mistake is highest.

What You Would Build in Taskade

Picture a competitor watch app for a small product team. Every Monday, you want to know what changed on five competitors' pricing and docs pages. An agent reads each site, prefers the llms.txt index and the Markdown copies where a site offers them, and falls back to the HTML page where it does not. It writes the changes into a project your team already uses, and an automation posts the summary to your team channel after a teammate approves it.

In Taskade you describe exactly that in one prompt, and Taskade Genesis builds the app with its data, agent, and automation in place. You also learn, week by week, which competitors are agent-ready, because those are the sites your agent reads most cleanly. Describe yours and start building →

Related Concepts

• MCP vs API: when an agent needs a protocol and when a direct call is enough
• Model Context Protocol (MCP): the action layer of an agent-ready site
• MCP Server: what the server side of MCP does
• MCP Server in Taskade: publish your workspace to outside agents
• Agentic Commerce: agent readiness applied to buying and selling
• Computer-Use Agents: agents that fall back to clicking a site built for people
• Agent-to-Agent Protocol (A2A): how agents find and call each other
• Tool Calling: how an agent invokes a tool on any server
• Prompt Injection: why an open action path is a risk
• Human-in-the-Loop: keep a person on the decisions that matter

Frequently Asked Questions About Agent-Ready Websites

What is an agent-ready website?

An agent-ready website is a site that AI agents can read and act on without parsing a layout built for people. It serves Markdown copies of its pages, publishes an llms.txt index, offers an MCP server or a documented API for actions, returns readable rate-limit headers, and gives the same facts on every surface.

What is the difference between llms.txt and a Markdown copy?

llms.txt is one index file at the site root that lists the pages worth reading, with a short note for each. A Markdown copy is the full text of one page at the page URL plus .md. The index tells an agent where to go, and the copy is what it reads when it gets there.

Does a Markdown copy hurt SEO?

Not when the copy sends a Link header with rel="canonical" that points at the HTML page. Search engines then treat the HTML page as the original and keep ranking it. Without that header, the copy can look like duplicate content.

Should I serve Markdown by content negotiation or at a separate URL?

A separate .md URL is safer. Content negotiation returns Markdown or HTML on the same URL based on the Accept header, and many CDN caches ignore that header, so the wrong version can be cached and served to the wrong reader. A distinct URL caches cleanly. If you also negotiate, make sure every cache varies on Accept.

Do I need an MCP server to be agent-ready?

Not to be readable. Markdown copies and llms.txt cover reading. You need an MCP server or a documented API when agents must act, for example to search a catalog, create a record, or check out. Require credentials for any action. See MCP vs API for when each fits.

How should rate limits work for AI agents?

Return HTTP 429 with headers that give the budget, the requests left, and the seconds until the window reopens. Key the budget to the account rather than the token, give heavy endpoints their own budget, and publish the rules so an agent can back off on schedule.

What does "APIs that lie" mean?

Guillermo Rauch used the phrase in an October 2026 interview for APIs that return wrong, stale, or inconsistent data, or break without a clear error. People work around them. Agents copy the wrong value and act on it. The fix is one answer per fact on every surface, real error codes, and a check that fails on a mismatch.

Is taskade.com agent-ready?

For its article sections, yes. Every /docs, /learn, and /blog article has a Markdown copy with a canonical link, the site publishes /llms.txt with section indexes, and paid plans include a hosted MCP server. The REST API returns 429 replies with budget and reset headers, and automated checks pin the Markdown routes and the agent entry points on every change. Sizing every rate limit for agent loops is ongoing work, as it is on most sites.

Further Reading

• The /llms.txt file: the proposal by Jeremy Howard, including the .md copy convention
• Vercel Ship 2026 recap: "The Third Customer" and agent experience
• Shopify Fall '26 Edition for developers: agent-ready stores by default
• Guillermo Rauch on Delphi: agents as customers and deleting agent-hostile tools
• The Taskade MCP server: connect your workspace to any MCP client
• The history of the Model Context Protocol: how MCP became the action layer
• OpenAPI to MCP code generators: turn an existing API into an MCP server
• Connect the hosted MCP server (Learn Taskade): step-by-step setup