NodpickWhat is ClaudeBot?
ClaudeBot is Anthropic's web crawler. It collects public web content that may contribute to training Anthropic's Claude models. Site owners control it through a ClaudeBot rule in robots.txt. Anthropic also operates Claude-SearchBot, which supports search results in Claude, and a user-triggered fetcher.
Training vs search
As with OpenAI, training and search are handled by different user agents. You can opt out of training and still allow the search crawler, so that Claude can find and cite your pages.
Check your setup
Review your robots.txt for ClaudeBot and Claude-SearchBot or a general rule that covers them. The free checker evaluates both.
Check which AI crawlers your site allowsFree AI crawler checker: GPTBot, ClaudeBot, PerplexityBot and more.Related terms
Be first when Nodpick opens
Nodpick opens soon. Join the waitlist for early access and the launch price.