Block AI Crawlers: robots.txt for GPTBot, Claude & CCBot
Emperola Editorial · · Updated · 4 min read
TL;DR
- AI crawlers honour robots.txt: block GPTBot, OAI-SearchBot, ClaudeBot, PerplexityBot, CCBot and Google-Extended by user agent.
- Block Google-Extended instead of Googlebot if you want to stay in Google Search but out of Gemini training.
- robots.txt is voluntary — Cloudflare's Block AI bots or a WAF rule is the only enforcement.
- Blocking stops future collection; it does not remove content already used for training.
AI crawlers collect public pages, blog posts, code and reviews to build training datasets. Blocking them is not about keeping Google out — it is about telling the bots behind ChatGPT, Gemini, Claude and Perplexity to stay away while Googlebot and Bingbot keep indexing your site normally.
This guide gives you copy-paste robots.txt rules, the Cloudflare switch that does the same without editing files, and the honest limits of each approach.
Who the AI Crawlers Are
- GPTBot — OpenAI, training data for ChatGPT models.
- OAI-SearchBot — OpenAI, indexing for ChatGPT Search. Blocking it removes your site from those answers.
- ChatGPT-User — user-initiated browsing, not training.
- Google-Extended — Google, controls Gemini training and grounding, separate from Googlebot.
- ClaudeBot and anthropic-ai — Anthropic, training data for Claude.
- PerplexityBot — Perplexity, indexing and answering.
- CCBot — Common Crawl, the archive many models are trained on indirectly.
- Meta-ExternalAgent — Meta, AI training.
- Bytespider — ByteDance, one of the most aggressive crawlers.
Two distinctions matter. Block Google-Extended, not Googlebot, if you want to stay in Google Search but out of Gemini training. And blocking GPTBot does not remove you from ChatGPT Search — that requires blocking OAI-SearchBot as well.
Step 1: Add Rules to robots.txt
robots.txt lives at https://yourdomain.com/robots.txt and must be publicly readable. Group the AI bots and disallow everything:
User-agent: GPTBot
Disallow: /
User-agent: OAI-SearchBot
Disallow: /
User-agent: Google-Extended
Disallow: /
User-agent: ClaudeBot
Disallow: /
User-agent: PerplexityBot
Disallow: /
User-agent: CCBot
Disallow: /
User-agent: Meta-ExternalAgent
Disallow: /
User-agent: Bytespider
Disallow: /
Keep your existing User-agent: * block. If it already says Disallow: /, you are blocking every bot including search engines — fix that first.
Step 2: Block at the Cloudflare Level
robots.txt is a request, not enforcement. If you use Cloudflare, you can refuse the request outright:
- Open Security → Bots and enable Block AI bots. This blocks verified AI crawlers at the edge and takes effect within minutes.
- On paid plans, AI Crawl Control adds per-bot analytics and allows selective blocking instead of a blanket rule.
- For anything not covered, add a WAF rule matching the user agent, for example
(http.user_agent contains "GPTBot")with the action Block.
Edge blocking is the only version that cannot be ignored, because the request never reaches your server.
Step 3: WordPress and Other CMS
- WordPress:
robots.txtis generated on the fly, so editing the file does nothing. Add the rules with a filter instead:
add_filter('robots_txt', function ($output) {
$bots = ['GPTBot','OAI-SearchBot','ClaudeBot','PerplexityBot','CCBot','Google-Extended','Meta-ExternalAgent','Bytespider'];
foreach ($bots as $bot) {
$output .= "\nUser-agent: {$bot}\nDisallow: /\n";
}
return $output;
}, 20);
- Static sites (Astro, Hugo, Next): edit the
robots.txtfile in your public folder. - Shopify, Wix, Squarespace: check whether the platform allows custom
robots.txt; if not, use the DNS or CDN layer.
Step 4: Check Your Server Logs
After two to four weeks, filter your access log for GPTBot, ClaudeBot, CCBot and PerplexityBot. You are looking for two things: whether they still arrive (a sign the rules are not being applied) and whether anything broke for Googlebot or Bingbot.
Common Crawl deserves a note: its snapshots are reused for years, so blocking CCBot today only affects future crawls. Your older content is already in the archive.
What robots.txt Does Not Do
- It is voluntary. OpenAI, Google and Anthropic honour it; scraper operators who do not publish a user agent simply ignore it.
- It does not delete collected data. Opting out stops future collection, not what already sits in a dataset.
- It does not stop spoofing. A scraper can send a normal browser user agent and bypass the list entirely.
- It does not cover AI browser extensions that fetch pages on behalf of a user.
That is why the practical combination is: robots.txt for the well-behaved bots, edge blocking for the rest, and minimal public data as the real protection. If your goal is your own data rather than your site, start with removing your personal data from the internet and with the AI scraping guide; if you want less tracking in general, see the private browser comparison.
FAQ
Does blocking GPTBot remove my data from ChatGPT?
No. Blocking stops future crawling; models trained on your content keep what they learned. To remove specific personal data you need a deletion request under GDPR or the provider's own form, and even then removal from model weights is limited.
What is the difference between Googlebot and Google-Extended?
Googlebot indexes pages for Google Search. Google-Extended is a separate control that decides whether your content may be used to train Gemini models. You can block Google-Extended and stay fully indexed in Search.
Does ChatGPT respect robots.txt?
OpenAI's published crawlers do — GPTBot for training and OAI-SearchBot for search. That is also why the two are separate: blocking one does not affect the other.
Does Cloudflare block AI bots by default?
No. The Block AI bots toggle is opt-in and covers verified crawlers only. Unverified scrapers need a WAF rule, rate limiting or bot management.
Can I block AI crawlers without editing robots.txt?
Yes — through Cloudflare's AI bot blocking, a WAF rule, or your hosting provider's bot controls. Editing robots.txt is still worth it: it is the only signal every provider's documentation says they read.