Blocking GPTBot does not remove you from ChatGPT's citations. What each AI crawler actually does, and what blocking it costs.
Most robots.txt advice about AI treats every bot the same: allow them all, or block them all. But AI crawlers do different jobs, and blocking one costs something very different from blocking another.
Three roles
| Role | What it does | Examples |
|---|---|---|
| Search | Fetches pages to answer a live question | OAI-SearchBot, Claude-SearchBot, PerplexityBot, Bingbot |
| User fetch | Fetches a page because a user asked about it | ChatGPT-User, Claude-User, Perplexity-User |
| Training | Collects pages for future model training | GPTBot, ClaudeBot, Google-Extended, CCBot |
What blocking each role costs
- Search. Block it and that assistant cannot cite your site in today's answers. This is the one you almost never want to block.
- User fetch. Block it and “summarise this page” requests fail, but discovery is unaffected.
- Training. Block it and you opt out of future training data. It does not remove you from today's citations.
A common configuration
Teams that want to stay citable while opting out of training often separate the two explicitly:
User-agent: OAI-SearchBot
Allow: /
User-agent: PerplexityBot
Allow: /
User-agent: GPTBot
Disallow: /Whether to opt out of training is a business decision. It is worth making deliberately — and separately from the search decision.
Check every important path, not just the homepage
The root path is the one almost nobody blocks, so testing / alone reports “open” for nearly every site. A robots.txt that allows / but disallows /pricing or /docs hides exactly the pages buyers are sent to. Test the URLs that matter.
Then make sure there is something to read
Search-role crawlers fetch HTML and do not run scripts. If your copy only appears after JavaScript hydrates the page, an allowed crawler still sees an empty shell. Server-render the words you want quoted.
Checklist
- 1List the AI crawlers in your robots.txt by role.
- 2Allow search-role crawlers on every buyer-facing path.
- 3Decide on training crawlers separately.
- 4Confirm key pages render their main content without JavaScript.