Choose which AI crawlers can visit your site and get ready-to-paste robots.txt rules, for example to block AI training while staying in AI search results.
Loading the tool…
Training, search and user crawlers
AI companies run different crawlers for different jobs, so you can block one without the others.
| Company | Training | Search | User requests |
|---|---|---|---|
| OpenAI | GPTBot | OAI-SearchBot | ChatGPT-User |
| Anthropic | ClaudeBot | Claude-SearchBot | Claude-User |
| Google-Extended | Googlebot (normal search) | – | |
| Perplexity | – | PerplexityBot | Perplexity-User |
| Apple | Applebot-Extended | Applebot | – |
Which option to choose
- Block training, allow search: keeps your content out of future model training while still letting AI assistants find and cite you. A popular middle ground.
- Block everything: maximum control, but you may vanish from AI answers.
- Allow everything: maximum visibility.
Limits
robots.txt is a voluntary standard. Major AI companies say they respect it, but not every crawler does, and visits triggered by a user’s question may not follow it. Blocking only applies to future crawling, not content already collected.
Frequently asked questions
Does blocking Google-Extended remove me from Google?
No. Google Search and AI Overviews use the normal Googlebot. Google-Extended only controls use in Gemini training and grounding.
How do I block ChatGPT?
Add User-agent: GPTBot with Disallow: / to stop training. Block OAI-SearchBot too if you don’t want to appear in ChatGPT search.
Where does robots.txt go?
In the root of your domain, at yourdomain.com/robots.txt, and on each subdomain.
Will this stop all scraping?
No. Use a firewall or bot protection for crawlers that ignore robots.txt.
Sources
- OpenAI: Overview of OpenAI crawlers
- Anthropic: Does Anthropic crawl the web and how can site owners block the crawler?
- Google: Common crawlers, including Google-Extended
Last updated: