What AI crawlers may do
AI services fetch pages with their own crawlers, either to answer questions with sources or to train models. In robots.txt you decide separately for each purpose who may access your pages.
How it works
A crawler is a program that fetches pages automatically. Each AI service sends its crawlers under a name of their own, the user agent. Broadly there are two kinds:
- Search agents fetch pages to answer a question and cite the page as a source:
OAI-SearchBotandChatGPT-User(OpenAI),Claude-SearchBotandClaude-User(Anthropic),PerplexityBotandPerplexity-User(Perplexity).GooglebotandBingbotbelong here too: Google's AI Overviews and Microsoft Copilot are built on their index. - Training crawlers collect pages that models are trained on:
GPTBot,ClaudeBot,Google-Extended,Applebot-Extended,CCBot(Common Crawl),Meta-ExternalAgentandBytespider.
What a crawler may do is set out in the file /robots.txt on your website. Each group starts with User-agent: and holds Allow and Disallow rules. If a group names a crawler explicitly, that crawler reads only this group; the group User-agent: * applies only to crawlers without a group of their own. Where several rules match, the longest wins, and on a tie Allow wins.
What matters
- Let the search agents in. If you block them, you won't appear as a source in the answers of ChatGPT, Claude or Perplexity. bytereport checks whether robots.txt lets these eight agents reach the start page: if it blocks some, bytereport warns; if it blocks all of them, that is a failure.
- Training is your choice. You can block training crawlers without losing search. This does not count towards the score. Models that know your pages from training are, however, more likely to name you without a web search.
Google-Extendedis not a crawler. The name only controls whether Google may use your content for its Gemini models. It has no effect on Google Search or AI Overviews.- Watch out for blanket blocks. A
Disallow: /underUser-agent: *blocks every crawler without its own group, the search agents included. - Stay indexable. A page that is allowed but marked
noindexis still missing from search; see indexability.
How to set it up
robots.txt sits in the root directory of the website. Many content management systems and SEO plugins generate it and offer a field for it; otherwise your agency or hosting provider changes it. These groups explicitly allow the search agents:
User-agent: OAI-SearchBot
Allow: /
User-agent: Claude-SearchBot
Allow: /
User-agent: PerplexityBot
Allow: /To rule out training, add groups of their own:
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: Google-Extended
Disallow: /Then check that a firewall does not turn the crawlers away regardless; see bot protection.