Can a firewall block AI crawlers?
Firewall bot blocking happens when a web application firewall or CDN stops AI crawlers even though robots.txt allows them. Cloudflare, for example, began blocking known AI crawlers by default for new domains in July 2025. If a bot receives an error or challenge, the assistant it serves cannot read your pages.
How to find out
Check the firewall or bot-management settings, and look in your server logs for 403 or 429 responses to bot user agents. Allow the search and user-triggered bots you want, and keep blocking the ones you do not.
Example: robots.txt allows OAI-SearchBot, but the CDN returns a 403 to it, so the pages are never read.
How does Nodpick help?
The free AI crawler checker reads a site's robots.txt. It cannot see firewall rules, so check those in your CDN settings. Run the free AI visibility check to see your own numbers; the method shows how they are made, and pricing lists the plans.
Check which AI crawlers your site allowsFree AI crawler checker: GPTBot, ClaudeBot, PerplexityBot and more.