Which AI crawler user agents handle training, search and user requests?

User agentPurpose
GPTBotCollects content that may be used to train OpenAI models.
OAI-SearchBotFinds pages for ChatGPT search.
ChatGPT-UserVisits pages for user actions; robots.txt rules may not apply.
ClaudeBotCollects web content that could contribute to model training.
Claude-SearchBotIndexes content to improve search responses.
Claude-UserRetrieves pages in response to user requests.
PerplexityBotFinds pages for Perplexity search results; not used to train AI models.
Perplexity-UserFetches a page when someone asks Perplexity a question; generally ignores robots.txt.
Applebot-ExtendedDoesn't crawl pages; disallowing it opts your content out of training Apple's generative AI models while leaving it eligible for Apple's search results.
CCBotCrawls for Common Crawl's open web archive, which anyone can use.
Your robots.txt blocks some AI crawlers

Block GPTBot in robots.txt while allowing ChatGPT search

User-agent: GPTBot
Disallow: /

User-agent: OAI-SearchBot
Allow: /

Block ClaudeBot and GPTBot without blocking AI search

  • Add these groups to your existing robots.txt. A crawler with its own group skips your User-agent: * rules, so copy any paths you keep closed to every crawler into the OAI-SearchBot group. OpenAI treats each setting as independent, and says a change can take about 24 hours to reach ChatGPT search.

  • To block ClaudeBot, add a User-agent: ClaudeBot group with Disallow: /. Choose separately whether to allow Claude-SearchBot and Claude-User; Anthropic's bot guide explains the effect of refusing each one.

Check whether Cloudflare is blocking AI crawlers

A permissive robots.txt doesn't override an edge block. In Cloudflare's AI crawler policies, Search, Agent and Training each get their own action, and new domains start with all three allowed. If Bot Preference Sync is on and a policy blocks traffic or disallows training, Cloudflare also adds its own rules to the top of your robots.txt, so the file crawlers read can differ from the one you deploy.

Frequently asked questions

Does blocking Google-Extended remove my site from Google Search?

No. Google-Extended controls certain Gemini training and grounding uses. It doesn't affect inclusion or rankings in Google Search, and it isn't a separate HTTP user agent. To keep your pages out of AI Overviews and AI Mode, use the Search generative AI control in Search Console.

  • AI visibility

    See whether ChatGPT and other AI tools can find your site at all.

  • llms.txt

    Learn what goes in an llms.txt file and which AI tools read one.

  • robots.txt

    Read crawler rules without mistaking a crawl block for removal from search.

  • JavaScript SEO

    Find an empty app shell and review whether the server sends useful page content.