AI Crawler robots.txt Generator
Choose exactly which AI bots can train on or search your store, bot by bot.
Your details
robots.txt rules for AI crawlers
# AI crawler rules. Add these groups to your existing robots.txt (don't replace the whole file).
User-agent: OAI-SearchBot
Allow: /
User-agent: ChatGPT-User
Allow: /
User-agent: GPTBot
Disallow: /
User-agent: Claude-SearchBot
Allow: /
User-agent: Claude-User
Allow: /
User-agent: ClaudeBot
Disallow: /
User-agent: PerplexityBot
Allow: /
User-agent: Perplexity-User
Allow: /
User-agent: Google-Extended
Disallow: /
User-agent: CCBot
Disallow: /
User-agent: Bytespider
Disallow: /
User-agent: Amazonbot
Disallow: /
User-agent: Applebot-Extended
Disallow: /
User-agent: Meta-ExternalAgent
Disallow: /
Shopify: paste these groups into templates/robots.txt.liquid (Themes → Edit code) alongside the default rules. WooCommerce/WordPress: paste them into your SEO plugin's robots.txt editor, or your site's physical robots.txt file.
Search bots and training bots aren't the same
AI companies generally run two kinds of crawler: one that collects content to train future models, and a separate one that fetches pages to answer a live search or chat query (or to cite a source). Blocking the first doesn't affect the second, they use different user-agent names on purpose so you can choose independently.
| Bot | Company | Purpose |
|---|---|---|
| GPTBot | OpenAI | Model training |
| OAI-SearchBot | OpenAI | ChatGPT search results |
| ChatGPT-User | OpenAI | Fetches a page a user asked ChatGPT to open |
| ClaudeBot | Anthropic | Model training |
| Claude-SearchBot | Anthropic | Claude search results |
| Claude-User | Anthropic | Fetches a page a user asked Claude to open |
| Google-Extended | Gemini training and AI Overviews grounding (not Search ranking) | |
| PerplexityBot | Perplexity | Search indexing |
| Perplexity-User | Perplexity | User-triggered fetch (Perplexity says this ignores robots.txt) |
| CCBot | Common Crawl | Training data used by many AI labs |
| Bytespider | ByteDance | Training and TikTok content features |
| Amazonbot | Amazon | Training |
| Applebot-Extended | Apple | Training (separate from Applebot, which is for Siri and Spotlight) |
| Meta-ExternalAgent | Meta | Training |
Blocking a training bot is a business choice, not an SEO or visibility risk: Google-Extended in particular only controls Gemini training and grounding, and Google has confirmed it doesn't affect Search ranking. Blocking a search bot like OAI-SearchBot can keep your store out of that AI product's answers.
Adding the rules
- 01
Shopify
Shopify themes can customize
robots.txt.liquid(in Themes → Edit code). Paste these groups in alongside the default ones, or use our Shopify robots.txt Generator if you also want to manage crawl-trap URLs like filtered collections at the same time. - 02
WooCommerce / WordPress
Most SEO plugins (Yoast, Rank Math) have a robots.txt editor under their settings. Paste these groups in there, or edit the physical robots.txt file at your site's root if you manage it directly.
- 03
Check it worked
Run the AI Crawler Checker against your homepage afterward.
Questions people ask
Does blocking these bots remove content already used for training?
No. robots.txt only controls future crawling. It has no effect on models already trained on your content.
Will blocking training bots hurt my Google ranking?
No. None of the training-only bots listed here (GPTBot, ClaudeBot, Google-Extended, CCBot, Bytespider, Applebot-Extended, Amazonbot, Meta-ExternalAgent) are used for Google Search ranking.
Some of these bots say they ignore robots.txt anyway. Why include them?
Blocking is still the documented, respectful way to ask; well-behaved crawlers honor it. A few companies have said their user-triggered "fetch this one page for me" agents may not check robots.txt, since a person, not a crawl job, asked for that specific page.
This list will go out of date, won't it?
Likely, new bots appear often. Re-run this generator every few months and compare against your current robots.txt.
Related tools
Want this builtinto your store?
I'm Emmanuel, a Shopify and WooCommerce developer. If this tool pointed at something you'd rather have fixed for you (a faster store, cleaner product data, a migration, a custom feature), send me a note.
Emmanuel Asika
Shopify & WooCommerce developer, Dublin
I build and fix online stores: new builds, migrations, speed, SEO and the custom bits apps can't do.
Last reviewed 30 September 2026.
Sources: OpenAI: bots used by ChatGPT and OpenAI · Anthropic: does Anthropic crawl data from the web? · Google: Google crawlers overview (Google-Extended)