How does robots.txt work for AI crawlers?
robots.txt is the plain-text file at the root of a domain that tells crawlers which paths they may fetch (RFC 9309). Each AI provider publishes its own crawler names, so you can allow search bots while blocking training bots. A rule that blocks a bot can remove you from that assistant's answers.
How do you write the rules?
Add a User-agent line per bot and Allow or Disallow rules below it. Training and search bots have different names, for example GPTBot and OAI-SearchBot at OpenAI, so decide for each one. The rules are requests, which well-behaved crawlers follow; they do not secure content.
Example: User-agent: GPTBot with Disallow: / keeps that training bot out, while User-agent: OAI-SearchBot with Allow: / lets the search bot read your pages.
Common mistakes
A blanket Disallow for all bots, a copied block list that includes search bots, and a firewall that blocks bots even though robots.txt allows them. Test the live file after every change.
How does Nodpick help?
The free AI crawler checker reads a site's robots.txt and shows which AI bots it allows. Run the free AI visibility check to see your own numbers; the method shows how they are made, and pricing lists the plans.
Check which AI crawlers your site allowsFree AI crawler checker: GPTBot, ClaudeBot, PerplexityBot and more.Read more: llms.txt: what it is and how to write one