What is the difference between training and search crawlers?
Training and search crawlers are the two kinds of AI bot: training crawlers such as GPTBot and ClaudeBot collect content for model training, while search crawlers such as OAI-SearchBot and PerplexityBot build the index that answers draw on. Blocking training does not remove you from search answers, but blocking the search bot can.
Why does the split matter?
Many sites want to keep their content out of model training and still be recommended in answers. With separate user agents that is possible: disallow the training bot and allow the search bot. A single blanket rule cannot do both.
Example: a publisher disallows GPTBot to stay out of training but allows OAI-SearchBot so ChatGPT search can still cite its articles.
Check before you copy a block list
Lists found online often mix both kinds. Each provider documents which bot does what: OpenAI (GPTBot, OAI-SearchBot, ChatGPT-User), Anthropic (ClaudeBot, Claude-SearchBot, Claude-User), Google (Googlebot, Google-Extended). Verify the current names in their docs.
Check which AI crawlers your site allowsFree AI crawler checker: GPTBot, ClaudeBot, PerplexityBot and more.