AI crawlers found our new sites in 75 minutes

Checked 26 October 2026.

Short answer: Within 75 minutes of launch, verified crawlers from OpenAI, Anthropic, Google and Bing had requested all three of our new sites. That shows that being found by machines is fast. It does not show that an assistant names you, and it does not show that anyone clicks. For that you have to look at the answers themselves.

What did we observe in the first week?

On 19 October 2026 at 16:06 UTC we launched three new sites on one server: a reference site about web crawlers, an analytics how-to site and a data site. We then counted requests for seven days, until 25 October 23:59 UTC.

A request counts as verified only if its IP address is in the operator's published list for that bot. Details are in the last section.

Bot	Operator	Purpose	First verified request after launch	Verified requests
OAI-SearchBot	OpenAI	Search index	43 min	2,309
Googlebot	Google	Search index	10 to 13 min	1,138
ClaudeBot	Anthropic	Model training	41 to 74 min	579
Applebot	Apple	Search and Apple features	19 to 21 h	502
GPTBot	OpenAI	Model training	43 min	363
Amazonbot	Amazon	Amazon services	3.8 and 4.2 days (2 of 3 sites)	325
ChatGPT-User	OpenAI	User-triggered fetch	26 to 43 h	167
Bingbot	Microsoft	Search index	about 4 min	120
PerplexityBot	Perplexity	Search index	28 h, 5.1 days, 6.0 days	15
Claude-User	Anthropic	User-triggered fetch	83 min (1 of 3 sites)	3
DuckAssistBot	DuckDuckGo	AI answers	4.1 days (1 of 3 sites)	2

Across the three sites that makes 5,523 verified requests from these eleven bots. Ranges span the three sites, each measured separately. Claude-SearchBot and Perplexity-User also publish IP lists, and we saw no verified request from either.

What does it mean when an index bot crawls you?

OpenAI says OAI-SearchBot is for ChatGPT search results, and that it is a separate setting from GPTBot, which it describes as relating to training. Anthropic describes Claude-SearchBot as navigating the web to improve search result quality. Perplexity describes PerplexityBot as designed to surface and link websites in Perplexity search results, and says it is not used to crawl content for AI foundation models.

So when OAI-SearchBot, PerplexityBot or Googlebot fetch your pages, you are in a position to be eligible for search-style answers. Eligible is the right word. Being fetched is a precondition, not a ranking or a promise. In our case OAI-SearchBot was by far the busiest bot (2,309 verified requests), while PerplexityBot was among the rarest (15), and on two of the three sites it needed days to arrive.

Is a training crawler a different decision?

Yes. GPTBot and ClaudeBot are the training-related crawlers. Anthropic says that when a site restricts ClaudeBot, it signals that the site's future material should be excluded from its model training datasets. OpenAI says a site can allow OAI-SearchBot to appear in search results while disallowing GPTBot.

That is a policy choice about training, not about visibility in search answers. Both reached our sites quickly (GPTBot after 43 minutes, ClaudeBot after 41 to 74), with 363 and 579 verified requests. Decide per bot, on purpose, instead of with one blanket rule. The AI crawler glossary entry lists the main bots.

What do user-triggered fetches tell you?

ChatGPT-User, Claude-User and Perplexity-User are fetches made while a person's conversation needs a page. OpenAI says ChatGPT-User may visit a web page when a user asks ChatGPT or a custom GPT a question, and that because the action is user-initiated, robots.txt rules may not apply. Anthropic describes Claude-User similarly. Perplexity says Perplexity-User may visit a page to answer a question and include a link to it.

In a server log, these are the closest signal that an assistant used your page for someone. They were rare and concentrated in our data:

• ChatGPT-User: 167 verified requests, but 151 of them were for the homepages of two sites, 107 on one homepage alone over five days. We cannot tell from logs why one homepage drew so many.
• Only 13 distinct deeper pages were fetched, each between one and four times.
• Claude-User: 3 requests on one site, 83 minutes after launch, for robots.txt, the homepage and the imprint. That pattern looks like a launch check, quite possibly our own. Perplexity-User: none verified.

A fetch does not tell you what the assistant wrote afterwards.

Does being read mean being named, and being named mean a click?

No, and the third step in that chain is visible in our own Search Console. For the 30 days to 25 October, the three sites together had 4 clicks from 948 impressions. Most pages sat at average positions far beyond page one, and one page at about position 7 had 0 clicks. These are new domains, so low numbers are expected. The contrast is the point: thousands of bot requests, a handful of clicks.

The chain has three separate steps:

1. Read: a bot fetched the page. Your logs show this.
2. Named: an assistant mentions your brand or cites your page in an answer. Logs cannot show this.
3. Clicked: a person follows the link. Analytics shows this, if the assistant sends a referrer. See how to track traffic from AI assistants.

For a brand, step 2 is the one that matters most, and it is the part no log file records. That is why brands need to measure mentions in the answers themselves, by asking the questions their buyers ask and recording what comes back.

Why verify a bot before you count or block it?

Because the user agent string is only a claim. In the same week, 32 ChatGPT-User requests and 34 OAI-SearchBot requests carried those names from IP addresses outside OpenAI's published lists.

Bot	Failed verification	Verified
Googlebot	65	1,138
OAI-SearchBot	34	2,309
PerplexityBot	42	15
ClaudeBot	32	579
ChatGPT-User	32	167
GPTBot	30	363
CCBot	28	0
Bingbot	24	120

PerplexityBot is the clearest case: 42 failed against 15 verified. CCBot, the Common Crawl bot, had 28 failed and none verified. Counting by user agent alone would have overstated both. Meta's meta-externalagent also appeared, but Meta publishes no IP list, so we could not verify it and left it out.

The same applies to allow and block rules: base IP-level rules on the operator's list, not on a name.

What should a small team do this week?

1. Decide robots.txt per bot type. Index bots (OAI-SearchBot, Claude-SearchBot, PerplexityBot, Googlebot, Bingbot) decide eligibility for search-style answers. Training bots (GPTBot, ClaudeBot) are a separate choice. Check what you currently allow with the AI crawler checker. Our llms.txt guide covers the optional file.
2. Look for user-triggered fetches in your logs. Filter for ChatGPT-User, Claude-User and Perplexity-User, verify the IPs against the published lists, and note which pages they request. If it is mostly the homepage, your deeper pages are not yet what assistants reach for.
3. Make key pages answer-first. Put the direct answer in the first lines and use clear headings, as described in answer-first content and our guide on getting cited by Perplexity and ChatGPT search.
4. Measure the answers. Pick 10 to 25 buyer questions, ask them in several assistants, and record who gets named. Repeat weekly. The AI visibility tracking guide shows a routine.

Nodpick measures exactly that step: whether and how assistants name your brand in their answers, which logs and referral reports cannot show. Nodpick is not open yet. The free check opens with the launch; join the waitlist.

How we measured, and the limits

• Sample: three sites, one server, one week (19 October 16:06 UTC to 25 October 23:59 UTC), all new domains. We make no claim about the wider web, older sites or other server setups.
• Verification: a request counts only if its IP is in the operator's published list (OpenAI, Anthropic, Google, Bing, Apple, Amazon, Perplexity, DuckDuckGo and Common Crawl lists). The lists were current when we ran the check. Meta publishes none.
• Launch time: first-request times are measured from 16:06 UTC on 19 October and are rounded. Bots that did not reach every site are marked in the table.
• Our own tests: when we look at our own sites through an assistant, the fetch looks like any other user-triggered fetch in the log. Some of the ChatGPT-User and Claude-User requests may therefore be ours.
• What we cannot see: why a homepage was fetched 107 times, what an assistant did with a fetched page, and whether a crawl led to any mention.

Frequently asked questions

How fast do AI crawlers find a new site? In our week, Bingbot and Googlebot arrived within about 13 minutes, and OAI-SearchBot and GPTBot after 43 minutes. Other bots took a day or more. It is a sample of three sites.

Does a crawl mean ChatGPT or Claude will name my brand? No. A crawl only makes pages available to the vendor's systems. Whether an assistant names you depends on the question and the answer it builds. You can only check that in the answers.

Should I block GPTBot or ClaudeBot? That is a training decision. OpenAI says OAI-SearchBot and GPTBot are independent settings, so you can allow one and disallow the other.

How do I tell a real bot from a fake one? Compare the request IP with the operator's published list. In our logs, between 24 and 65 requests per bot name came from outside those lists, and CCBot had none that verified.

Can I see AI assistants using my pages in my logs? Partly. User-triggered fetchers such as ChatGPT-User show that a page was fetched during a conversation. They do not show what was said, and they were rare in our data.

Sources

All read on 26 October 2026.

• OpenAI, Overview of OpenAI crawlers (OAI-SearchBot, GPTBot, ChatGPT-User): https://platform.openai.com/docs/bots
• Anthropic, Does Anthropic crawl data from the web, and how can site owners block the crawler? (ClaudeBot, Claude-User, Claude-SearchBot): https://support.claude.com/en/articles/8896518-does-anthropic-crawl-data-from-the-web-and-how-can-site-owners-block-the-crawler
• Perplexity, Perplexity Crawlers (PerplexityBot, Perplexity-User): https://docs.perplexity.ai/guides/bots
• Google Search Central, Overview of Google crawlers and fetchers (last updated 2026-06-12 UTC): https://developers.google.com/search/docs/crawling-indexing/overview-google-crawlers
• Own server logs and Search Console, 19 to 25 October 2026 (aggregates).