NodpickWhat is AI crawler?
Also: AI bot, LLM crawler
An AI crawler is an automated program that fetches web pages for an AI company, either to train models, to build a search index for an assistant, or to open a page when a user asks. Each identifies itself with a user-agent name you can allow or block in robots.txt.
Three purposes
- Training: collecting content that may be used to train models (for example GPTBot, ClaudeBot, CCBot).
- Search: indexing pages so an assistant can retrieve them (for example OAI-SearchBot, PerplexityBot).
- User requests: fetching a page a user asked the assistant to open (for example ChatGPT-User).
Blocking and visibility
Blocking training crawlers is a business decision about your content. Blocking search crawlers can keep you out of the assistant's results. Many sites block everything by accident, so check your robots.txt.
Check which AI crawlers your site allowsFree AI crawler checker: GPTBot, ClaudeBot, PerplexityBot and more.Related terms
Compare tools
Be first when Nodpick opens
Nodpick opens soon. Join the waitlist for early access and the launch price.