Decide which AI crawlers to block and which to allow
AI crawlers serve different purposes. A rule that refuses model training can leave search access open, while a rule aimed at a search crawler can keep your content out of that service's answers.
Which AI crawler user agents handle training, search and user requests?
| User agent | Purpose |
|---|---|
| GPTBot | Collects content that may be used to train OpenAI models. |
| OAI-SearchBot | Finds pages for ChatGPT search. |
| ChatGPT-User | Visits pages for user actions; robots.txt rules may not apply. |
| ClaudeBot | Collects web content that could contribute to model training. |
| Claude-SearchBot | Indexes content to improve search responses. |
| Claude-User | Retrieves pages in response to user requests. |
| PerplexityBot | Finds pages for Perplexity search results; not used to train AI models. |
| Perplexity-User | Fetches a page when someone asks Perplexity a question; generally ignores robots.txt. |
| Applebot-Extended | Doesn't crawl pages; disallowing it opts your content out of training Apple's generative AI models while leaving it eligible for Apple's search results. |
| CCBot | Crawls for Common Crawl's open web archive, which anyone can use. |

Block GPTBot in robots.txt while allowing ChatGPT search
User-agent: GPTBot
Disallow: /
User-agent: OAI-SearchBot
Allow: /
Block ClaudeBot and GPTBot without blocking AI search
Add these groups to your existing
robots.txt. A crawler with its own group skips yourUser-agent: *rules, so copy any paths you keep closed to every crawler into theOAI-SearchBotgroup. OpenAI treats each setting as independent, and says a change can take about 24 hours to reach ChatGPT search.To block ClaudeBot, add a
User-agent: ClaudeBotgroup withDisallow: /. Choose separately whether to allowClaude-SearchBotandClaude-User; Anthropic's bot guide explains the effect of refusing each one.
Check whether Cloudflare is blocking AI crawlers
A permissive robots.txt doesn't override an edge block. In Cloudflare's AI crawler policies, Search, Agent and Training each get their own action, and new domains start with all three allowed. If Bot Preference Sync is on and a policy blocks traffic or disallows training, Cloudflare also adds its own rules to the top of your robots.txt, so the file crawlers read can differ from the one you deploy.
Frequently asked questions
Does blocking Google-Extended remove my site from Google Search?
No. Google-Extended controls certain Gemini training and grounding uses. It doesn't affect inclusion or rankings in Google Search, and it isn't a separate HTTP user agent. To keep your pages out of AI Overviews and AI Mode, use the Search generative AI control in Search Console.
Review robots.txt and content AI crawlers can read
AI visibility
See whether ChatGPT and other AI tools can find your site at all.
llms.txt
Learn what goes in an llms.txt file and which AI tools read one.
robots.txt
Read crawler rules without mistaking a crawl block for removal from search.
JavaScript SEO
Find an empty app shell and review whether the server sends useful page content.