Blocking GPTBot does not remove you from ChatGPT's citations. What each AI crawler actually does, and what blocking it costs.
Most robots.txt advice about AI treats every bot the same: allow them all, or block them all. But AI crawlers do different jobs, and blocking one costs something very different from blocking another.
What are the three kinds of AI crawler?
| Role | What it does | Examples |
|---|---|---|
| Search | Fetches pages to answer a live question | OAI-SearchBot, Claude-SearchBot, PerplexityBot, Bingbot |
| User fetch | Fetches a page because a user asked about it | ChatGPT-User, Claude-User, Perplexity-User |
| Training | Collects pages for future model training | GPTBot, ClaudeBot, Google-Extended, CCBot |
Each company documents its own crawlers and what they are for: OpenAI, Anthropic and Perplexity. Check the user-agent names there before you write rules for them.
What does blocking each kind of crawler cost?
- Search. Block it and that assistant cannot cite your site in today's answers. This is the one you almost never want to block.
- User fetch. Block it and “summarise this page” requests fail, but discovery is unaffected.
- Training. Block it and you opt out of future training data. It does not remove you from today's citations.
How do you stay citable but opt out of training?
Teams that want to stay citable while opting out of training often separate the two explicitly:
User-agent: OAI-SearchBot
Allow: /
User-agent: PerplexityBot
Allow: /
User-agent: GPTBot
Disallow: /Whether to opt out of training is a business decision. It is worth making deliberately — and separately from the search decision.
Why check every important path, not just the homepage?
The root path is the one almost nobody blocks, so testing / alone reports “open” for nearly every site. A robots.txt that allows / but disallows /pricing or /docs hides exactly the pages buyers are sent to. Test the URLs that matter.
Can an allowed crawler actually read the page?
Search-role crawlers fetch HTML and do not run scripts. If your copy only appears after JavaScript hydrates the page, an allowed crawler still sees an empty shell. Server-render the words you want quoted.
How does Vidrys score AI crawler access?
The Vidrys answer-engine audit checks 15 AI crawlers: 4 search crawlers, 3 user-fetch crawlers and 8 training crawlers. It tests each one against your site's real URLs, not only the homepage, and weights what a crawler can reach by its role: 1 for search, 0.5 for user fetch and 0.25 for training. Crawler access is worth 12 of the audit's 100 points, and server-rendered content another 10. Site audits are included on every plan, from 50/mo on Solo at ₹999 a month (prices in INR, excluding GST, checked on 27 October 2026).
Checklist
- 1List the AI crawlers in your robots.txt by role.
- 2Allow search-role crawlers on every buyer-facing path.
- 3Decide on training crawlers separately.
- 4Confirm key pages render their main content without JavaScript.
Written by the Vidrys team. Published 13 Sept 2026.