# What agents take from an agent-first site, and what they send back

> Quynh Tran-Thanh tests what small businesses argue about when they build for AI agents (does ChatGPT send customers, do agents read Markdown, does llms.txt do anything, is an MCP server worth it) against six weeks of Anchor Terminal's own traffic and the changes we shipped in that time.

- Canonical: https://www.anchorterminal.com/blog/what-agents-send-back
- Markdown: https://www.anchorterminal.com/blog/what-agents-send-back.md (~4,550 tokens)
- Slim: https://www.anchorterminal.com/blog/what-agents-send-back.min.md (~1,980 tokens, same facts, less prose, for token-sensitive contexts)
- JSON: https://www.anchorterminal.com/blog/what-agents-send-back.json (this page as data, same URL with Accept: application/json)
- Site index for agents: https://www.anchorterminal.com/llms.txt (the key docs and every listing in one file, at most 16 MB, about 4 million tokens: https://www.anchorterminal.com/llms-full.txt)
- API: https://www.anchorterminal.com/api/v1/index.json
- MCP: https://www.anchorterminal.com/mcp (the whole catalogue as tools, no key)
- DNS: dig +short TXT <slug>.grade.anchorterminal.com (the grade as a DNS TXT record, for a sandbox with no HTTP where names still resolve; also top., best.<category>. and <a>.vs.<b>. under the zone, and the same under grade.letme.dev)
- Licence: CC BY 4.0, credit Anchor Terminal and link this page (https://www.anchorterminal.com/terms/#data)
- Updated: 2026-11-20

By Quynh Tran-Thanh, 20 November 2026. 2,184 words, about 9 min to read.

If you run a small business in 2026, someone has told you to get ready for agents. Write an llms.txt. Give every page a Markdown copy. Build an MCP server. Make sure ChatGPT can read you, because that's where your customers will come from.

We did all of it. Anchor Terminal was built for agents from the first day, and in October we wrote a guide telling other sites to do the same [[1]](#ref-1). So I wanted to check it against what actually happened on our server. What do agents take from a site built for them, and what do they send back?

## The research

I started with what people argue about. On Hacker News the same questions come round every few weeks, and they rarely come with data.

- Does ChatGPT send customers? "Tons of people click from ChatGPT! It's now the top referrer on most of my websites" [[2]](#ref-2)
- Does anyone read llms.txt? "LLMs are not reading llms.txt nor AGENTS.md files from servers" [[3]](#ref-3)
- Do agents want Markdown? One person logged the Accept header of ChatGPT's, Claude's and Perplexity's fetchers and wrote "None of them use text/markdown." [[4]](#ref-4)
- Is an MCP server worth building? "Why on earth would you make an MCP to let agents query your docs specifically?" [[5]](#ref-5)

There is some data on the first one. Cloudflare counted how many pages each AI company's bots fetch for each visitor its product sends back, across its whole network, and found about 50,000 for Anthropic, 887 for OpenAI and 118 for Perplexity in August 2025 [[6]](#ref-6). Ahrefs found AI chatbots sent 0.28 per cent of the traffic to 74,752 sites in March 2026 [[7]](#ref-7). Those are averages over millions of sites. Nobody I found had the numbers for one small site that did everything the guides say.

So I used ours. Every request to anchorterminal.com reaches our own Go server, which writes a line for it, and once a day the lines are summed into a report we read on our admin page [[8]](#ref-8). I took the 46 days from 5 October to 19 November 2026, with the detailed lists (pages, formats, who referred whom) for the last 30 [[8]](#ref-8).

- 1,197,467: visitors whose browser said it was a browser
- 11,992: of them that ran our page script, the people
- 285,419: crawler visitors
- 16,256: agent visitors

Anchor Terminal's traffic, 5 October to 19 November 2026. Source [8].

The server sorts every visitor by what it says it is. A crawler fetches pages for a search engine or an AI index, such as Googlebot, Applebot or GPTBot. An agent fetches because a person asked something right now, such as ChatGPT or Claude fetching for a user, or an MCP client [[9]](#ref-9). And a person is a browser that ran our page script. Before looking, I wrote down what I expected for each question, so I could see where I was wrong [[10]](#ref-10).

I also lined the numbers up against the changes we shipped in those six weeks, since we changed a lot. The ones that matter here are the MCP server going into the official registry and the app stores on 31 October, and the sitemaps we cut and reopened [[11]](#ref-11).

## What the data says

### The assistants read a lot and send almost nobody

For each company I counted its bots' requests over the 30 days and the visitors its product sent back [[8]](#ref-8). The AI companies read far more of our site than their averages across Cloudflare's network would suggest, and sent back less.

| Company | Its bots' requests | Visitors it sent | Requests a visitor |
| --- | --- | --- | --- |
| OpenAI (GPTBot, ChatGPT fetching for users) | 97,336 | 21 | 4,635 |
| Perplexity (PerplexityBot) | 135,338 | 16 | 8,459 |
| Microsoft (Bingbot) | 41,047 | 94 | 437 |
| Apple (Applebot) | 198,247 | 0 | none sent |
| Google (Googlebot) | 4,323 | 19,150 | 0.23 |

Source [[8]](#ref-8). Anthropic's fetchers came as 1,922 visitors and claude.ai sent 9 visitors back, but our report doesn't keep Anthropic's request count [[8]](#ref-8).

OpenAI's ratio is five times Cloudflare's 887, and Perplexity's is seventy times its 118 [[6]](#ref-6) [[8]](#ref-8). PerplexityBot made its 135,338 requests from just 258 addresses [[8]](#ref-8). Google is the one that runs the other way, with 4.4 visits from google.com for every page Googlebot fetched [[8]](#ref-8). AI assistants were 0.25 per cent of the visits that came with a referrer, almost exactly Ahrefs' 0.28 [[7]](#ref-7) [[8]](#ref-8).

That's a little unfair to the AI side, and I want to say how. Chat apps on a phone often strip where a visitor came from, so some of their visitors arrive looking like they came from nowhere. I counted AI referrals by request and Google's by visit, which flatters AI too. Even if the real AI number were ten times what we can see, Google would still send more.

I think this is the nature of an answer engine. An assistant reads a directory like ours to write its answer, and the answer is the product. The person gets what they needed without clicking. That's good for them and fine for us, since we built the site to be read by agents. But if you're building for agents because you hope ChatGPT will send you customers, our numbers say not yet.

### Who reads the Markdown depends on who you call an agent

Every listing, comparison and category page on our site has a Markdown copy, linked from the page itself, and llms.txt tells agents to append .md to any address to get it [[15]](#ref-15). We've said that agents use it, and in October they did. Agents that name themselves (ChatGPT and Claude fetching for a user, MCP clients, scripts) took the Markdown for 37 per cent of the pages that have a copy from 21 October to 12 November [[8]](#ref-8). From 13 to 19 November it was 16 per cent [[8]](#ref-8).

Comparisons are where agents take the Markdown most. Those are the pages that answer "X or Y", which is what a person asks an agent and what an agent searches for. On the home page it's under one per cent. One busy agent can move a day's share a long way, so I wouldn't read much into a single week.

But the agents that name themselves are a small part of who reads the Markdown. We served 855,197 Markdown copies in the 30 days [[8]](#ref-8). Crawlers took 68.8 per cent of them, clients that called themselves browsers and never ran our script took 28.7 per cent, and the named agents 2.5 per cent [[8]](#ref-8). A browser doesn't ask for a Markdown file by itself, so that middle group is something automated sending a browser's name.

The clearest sign came from a bug fix. Until 8 November, /tools/exa-mcp.md worked and /tools/exa-mcp/index.md was a 404, and llms.txt's instruction to append .md can be read either way [[11]](#ref-11) [[15]](#ref-15). We made both spellings work. Markdown requests from those browser-named clients went from about 3,600 a day (1 to 8 November) to about 17,000 (9 to 19 November) [[8]](#ref-8). Something had been following the instruction, guessing the other spelling and getting nothing. I think a lot of it is agents that don't say they're agents, but our data can't tell them apart from scrapers.

### Files for agents are barely read, a listing where agents look is

llms.txt, llms-full.txt, our /agents page and the files under /.well-known/ together were 0.43 per cent of agents' requests, and 0.05 per cent of crawlers' [[8]](#ref-8). That matches SE Ranking, which found no effect of llms.txt on how often nearly 300,000 domains were cited [[12]](#ref-12). Google says you don't need special files for its AI features at all [[13]](#ref-13).

The files agents did look for and didn't find were more telling. The most requested missing files were the OAuth discovery addresses an MCP client checks before it connects, then agent-card.json, agent.json and ai-plugin.json [[8]](#ref-8). Agents look for how to connect, not for a description.

The change that moved agent traffic was the MCP registry. On 31 October we published our MCP server to the official registry and filled in the app stores' forms [[11]](#ref-11). The next day MCP requests went from about 3 a day to about 3,000, and stayed there [[8]](#ref-8).

A day, before and after the MCP registry entry on 31 October

| Measure | 5 to 30 October | 1 to 19 November |
| --- | --- | --- |
| MCP requests | 3 | 3076 |
| Agent visitors | 234 | 514 |

Each series has a zero-based scale.

Agent visitors more than doubled and agents' downloads went from about 1 GB a day to 4.6 GB [[8]](#ref-8). An agent took 603 KB a request on average, against 47 KB for a person [[8]](#ref-8). There's a catch I didn't expect. Agents' requests for ordinary pages fell, from a median of 2,645 a day to 1,713 [[8]](#ref-8). My guess is that some agents that used to read the pages now ask the MCP server instead. Our data can't say how many of those MCP requests were real questions and how many were a client listing our tools or a registry checking we're up.

### Google is still the one that sends people, and one mistake cost most of it

This one we learnt the hard way. We resubmitted our sitemap to Google with too many pages in it, and Google stopped indexing us. Visits from google.com went from about 826 a day (1 to 12 November) to about 53 a day (13 to 19 November) [[8]](#ref-8). We've since resubmitted a smaller set.

Two things in the data stood out. Googlebot was only fetching about 100 to 200 pages a day all along, the smallest crawl of any big crawler on our site, so a young domain doesn't get much of Google's attention to waste [[8]](#ref-8). And over the same week the number of people who ran our script went up, from 248 a day to 463, because the rest of our readers come from Reddit, X and links [[8]](#ref-8). But Google sent more visitors than every AI assistant together, by hundreds of times, and a sitemap was enough to lose most of it.

### Most of your visitors aren't people

Of the visitors whose browser said it was a browser, a median of 0.96 per cent a day ran our page script [[8]](#ref-8). Singapore had 226,098 "browser" visitors in 30 days and 341 that ran the script [[8]](#ref-8). Phones were 2.6 per cent, which no real audience looks like.

Names lie too. Several crawlers publish the addresses they crawl from, so we check each request against them. One request in nine claiming to be Googlebot came from somewhere else, against one in two hundred for ChatGPT's fetcher [[8]](#ref-8). That's what people on Hacker News complain about. "Fake Googlebot visits are #1 in website logs I've been working on." [[14]](#ref-14)

## Key takeaways

- AI assistants read a site built for them thousands of times for every visitor they send. Google sent us more visitors than every AI assistant together.
- Agents that name themselves took the Markdown for about a third of pages in October and a sixth in mid November. Most of the Markdown went to crawlers and to clients calling themselves browsers.
- llms.txt and the other files for agents were under half a per cent of agents' requests.
- Being listed in the MCP registry took agent traffic from a trickle to thousands of requests a day overnight.
- One sitemap with too many pages cost us most of our Google visits in a week.
- Only about one in a hundred "browsers" in our logs was a person.

## What it means if you want agents to find your business

Build for agents because they read you, not because they'll send you customers. When an assistant reads your page, it's your page that shapes its answer, and on our numbers that happens thousands of times for each click. Count it as a channel of its own, and keep the search engine that still sends people in good health.

Go where agents look for tools. If you have an API or an MCP server, list it in the official MCP registry and the app stores before you write another file for agents. It was the only change of ours that moved agent traffic overnight.

Make the HTML readable and keep the Markdown. The agents that name themselves take the HTML more often than not, so the page itself matters most. But a Markdown copy is cheap, it's what was read on the comparison pages, and something follows the instructions you give for finding it. Make every address an agent might guess for it work, because a 404 there is invisible to you unless you look.

Be careful with Google. A small, clean sitemap of the pages that matter beat a big one for us, and the mistake was quick to make and slow to undo.

Count people by what they do, not by what they say they are. If your analytics count every browser as a person, most of your audience is scrapers. A script that runs, or a page kept in view for a few seconds, is a better test. And check that a Googlebot is Google before you let it past your limits.

What I couldn't settle is whether agents that read you without sending anyone still bring you business. A person who asks ChatGPT which tool to use and gets your name may sign up later without ever clicking a link we can see. Ahrefs found AI visitors were 0.5 per cent of its own visitors and 12.1 per cent of its signups [[7]](#ref-7). That's one site, and so is ours. We'll keep counting.

---

More in [From the data](https://www.anchorterminal.com/blog/from-the-data/index.md) · Previous from the data [Two handshakes. Who funds the agent stack, and whether it shows in the tools](https://www.anchorterminal.com/blog/two-handshakes-who-funds-the-agent-stack.md)

- Older: [OpenAI's agents now finish half its two-to-four-hour research tasks](https://www.anchorterminal.com/blog/openai-agents-half-the-research.md)
- Newer: [What to build that lasts. The jobs agents ask for that nobody finishes](https://www.anchorterminal.com/blog/the-jobs-nobody-finishes.md)
- All posts: https://www.anchorterminal.com/blog/index.md

Anchor Terminal grades the tools AI agents run on, with a source for every score. [The top tools](https://www.anchorterminal.com/top/index.md) · [Categories](https://www.anchorterminal.com/categories/index.md) · [Compare two tools](https://www.anchorterminal.com/compare/index.md) · [Prices](https://www.anchorterminal.com/prices/index.md) · [Sunsets](https://www.anchorterminal.com/sunsets/index.md) · [API, MCP server and CLI](https://www.anchorterminal.com/agents/index.md)

## References

[1] Anchor Terminal, "How to make your site and tool ready for agents", 15 October 2026. Source for our advice to write an llms.txt and a Markdown copy of every page. https://www.anchorterminal.com/blog/how-to-make-your-site-ready-for-agents

[2] Hacker News, comment by timpera, 13 October 2026. Source for the quote on ChatGPT as the top referrer. https://news.ycombinator.com/item?id=50078893

[3] Hacker News, comment by reconnecting, 18 February 2026. Source for the quote on llms.txt not being read. https://news.ycombinator.com/item?id=47058870

[4] Hacker News, comment by startages, 20 April 2026. Source for the quote on the fetchers' Accept header. https://news.ycombinator.com/item?id=47836721

[5] Hacker News, comment by drbscl, 20 November 2026. Source for the quote on building an MCP server for docs. https://news.ycombinator.com/item?id=50575206

[6] Cloudflare, "A deeper look at AI crawlers: breaking down traffic by purpose and industry", 28 August 2025. Source for the crawl-to-refer ratios of about 50,000 for Anthropic, 887 for OpenAI and 118 for Perplexity. https://blog.cloudflare.com/ai-crawler-traffic-by-purpose-and-industry/

[7] Ahrefs, "AI Chatbot Traffic: What It Is, and How to Get More", 15 May 2026. Source for AI chatbots' 0.28 per cent of traffic across 74,752 sites, and AI visitors' 0.5 per cent of Ahrefs' visitors and 12.1 per cent of its signups. https://ahrefs.com/blog/ai-chatbot-traffic/

[8] Anchor Terminal, the server's daily traffic summaries on the admin page, 5 October to 19 November 2026, read on 20 November 2026, not published. Source for every count of visitors, requests, formats, referrers, verified crawlers, megabytes and missing files in this post, worked out by analyse.py in the research folder.

[9] Anchor Terminal, "How to build an agent-first website, from our first six weeks and 5 million page views", 19 November 2026. Source for how we tell people, agents and crawlers apart. https://www.anchorterminal.com/blog/how-to-build-an-agent-first-website

[10] Anchor Terminal, the hypotheses for this post, written and committed on 20 November 2026 before the analysis ran, not published. Source for what I expected.

[11] Anchor Terminal, deploy notes and commits of 31 October, 8 November and 15 November 2026, not published. Source for the MCP registry entry and the app stores' forms on 31 October, both spellings of a Markdown address working from 8 November, and the sitemaps.

[12] SE Ranking, "LLMs.txt: Why Brands Rely On It and Why It Doesn’t Work", 15 October 2026. Source for 10.13 per cent of nearly 300,000 domains having the file and no effect on citations. https://seranking.com/blog/llms-txt/

[13] Google Search Central, "AI Features and Your Website", updated 10 December 2025. Source for Google saying no machine-readable or AI text files are needed. https://developers.google.com/search/docs/appearance/ai-features

[14] Hacker News, comment by ChillyCapy, 12 August 2026. Source for the quote on fake Googlebot visits. https://news.ycombinator.com/item?id=49276399

[15] Anchor Terminal, llms.txt, read on 20 November 2026. Source for the instruction to append .md to an address, or send Accept: text/markdown, to get a page's Markdown copy. https://www.anchorterminal.com/llms.txt

---

Query this catalogue directly over MCP: https://www.anchorterminal.com/mcp (Streamable HTTP, no key). How: https://www.anchorterminal.com/agents/#mcp
