How bot protection locks out AI crawlers
Firewalls and CDNs often block AI crawlers by rule, even when robots.txt allows them. The crawler then gets an error or a challenge page instead of your content, and your site is missing from AI answers.
How it works
Many websites sit behind a web application firewall (WAF) or a CDN (content delivery network), such as Cloudflare, Akamai or the protection of the hosting provider. These services check every request before it reaches your server. If a request looks automated, it is turned away or gets a challenge page that a human has to solve in a browser first.
A crawler cannot solve such a page. It sees “Just a moment…” or a captcha instead of your text. robots.txt does not help here: it is a request to the crawler, whereas the firewall decides beforehand and independently of it.
Often a single switch is behind it, such as “block AI crawlers”, or a rule against certain user agents. It then usually blocks every AI crawler, the search agents included.
What matters
- The same answer as for browsers. bytereport fetches the start page with the user agent of
OAI-SearchBot,PerplexityBotandClaudeBotand compares the result with a fetch as a browser. A different status or no answer counts as a block and as a failure. - No challenge page and no stub. A challenge page, or much less text (less than half the words, when the browser sees at least 150), counts as a warning.
- Exempt the search agents. If you don't want training, control that in robots.txt or block the training crawlers specifically, not every bot across the board.
- Recognise real crawlers. A user agent can be faked. Many services therefore publish the IP ranges of their crawlers, and some firewalls keep lists of verified bots. Such services may answer the real crawler differently from the test; the check shows what a rule on the user agent does.
How to set it up
The bot rules live in the dashboard of your CDN or firewall, or in the security settings of your hosting provider; if your agency set them up, it can change them. Check there:
- A switch such as “Block AI bots” or a similar setting against AI crawlers: let the search agents through where possible.
- Custom rules aimed at user agents such as
GPTBot,ClaudeBotor at “bot” in general. - Bot fight mode or a high security level that shows every unknown visitor a challenge.
Afterwards block only what you really don't want and control the rest in robots.txt; see AI crawlers. Then run the check again.