How Claude finds, cites and recommends brands
Claude searches the web when it helps the question, and every answer that uses web search includes citations. Anthropic runs three crawlers: Claude-SearchBot for search, Claude-User for user requests and ClaudeBot for training. To be found and cited, allow the first two. Blocking ClaudeBot only keeps your future content out of training.
Updated
How Claude decides to search
Anthropic describes web search as extending Claude's knowledge with real-time data: when a question benefits from current information, Claude calls a search tool and grounds its answer in content from the live web, and every such answer includes citations. In the newer Claude experience there is no switch; Claude searches when it helps. Questions about which tool to buy or which provider to hire usually qualify.
Three bots, three decisions
Anthropic separates search, user requests and training. Blocking Claude-SearchBot prevents your content from being indexed for search and, in Anthropic's words, may reduce your visibility and accuracy in user search results. Blocking Claude-User stops Claude from fetching your pages when a user asks. Blocking ClaudeBot signals that your future content should be excluded from training. Many sites block all three with one rule without meaning to; check your robots.txt and your CDN's AI-bot setting.
Recommended or only mentioned?
Without search, Claude answers from model knowledge, which lags behind and favors brands that are described widely and consistently. With search, the cited pages decide. If Claude names you but not as its pick, look at the cited pages: they usually describe a competitor as the fit for the exact situation in the question.
Which crawlers matter
- Claude-SearchBot
Crawls the web to improve the quality of Claude's search results.
If you block it: Your content is not indexed for search; Anthropic says this may reduce your visibility and accuracy in user search results.
- Claude-User
Fetches pages when a person asks Claude a question.
If you block it: Claude cannot retrieve your content in response to a user query, which may reduce your visibility for user-directed web search.
- ClaudeBot
Collects public web content that may be used to train Anthropic's models.
If you block it: Your future content is excluded from training data. It is not the switch for search.
What to do, by evidence
Each step carries an evidence grade, as in every report: A documented by the vendor, B backed by a study, C untested. Start with A and B.
- A · documented: The vendor's own documentation describes this.Source: Anthropic: Does Anthropic crawl data from the web?
Allow Claude-SearchBot and Claude-User in robots.txt and in your CDN's bot settings. Anthropic honors robots.txt and supports Crawl-delay if you need to slow it down.
- B · studied: An experiment or a large study of AI answers backs it. Not guaranteed for your brand.Source: AI search source study (Chen et al., 2025)
Get onto the third-party pages Claude cites for your questions. A large study found Claude, like the other assistants, cites mostly third-party (earned) sources.
- B · studied: An experiment or a large study of AI answers backs it. Not guaranteed for your brand.Source: GEO study (Aggarwal et al., 2023)
Make cited pages concrete: numbers, prices, named sources. The GEO study measured more visibility in generative answers for these edits.
- C · untested: No documented or studied effect yet. Optional.
Block ClaudeBot only if you do not want training use; it does not change search. There is no study of whether training opt-outs change later recommendations.
What Nodpick measures in Claude
- Whether the answer names your brand, and in which position among the brands it names.
- Whether it recommends you, only mentions you or warns against you, with a recommendation rate once there are enough answers.
- Who is named instead of you, and your share of voice against them.
- Which domains Claude cites for your questions, and whether your own pages are among them.
- Claude is in the free check and in every paid plan.
Common questions
Does blocking ClaudeBot remove me from Claude's answers?
Not from search. ClaudeBot is Anthropic's training crawler. Search uses Claude-SearchBot, and user requests use Claude-User. Each has its own robots.txt rule.
Does Claude cite its sources?
Yes, when it uses web search. Anthropic says every response that uses web search includes citations, so you can check which pages shaped the answer.
When does Claude search the web?
When a question benefits from current information, according to Anthropic. In the newer Claude experience it decides itself; users can also ask it to search.
Does Anthropic respect robots.txt?
Yes. Anthropic says its bots honor standard robots.txt directives and support the non-standard Crawl-delay extension.
Sources
Vendor documentation and studies, checked November 1, 2026.