Shopify robots.txt Generator
Build a robots.txt.liquid that keeps Shopify's default rules and adds yours for search pages, filters and AI crawlers.
Your details
Your templates/robots.txt.liquid
{%- comment -%}
Custom robots.txt. Shopify's default rules come from robots.default_groups and stay up to date.
The lines below only add to them.
{%- endcomment -%}
{%- capture extra_rules -%}
Disallow: /search
Disallow: /*/search?
Disallow: /collections/vendors?
Disallow: /collections/types?
Disallow: /*/collections/vendors?
Disallow: /*/collections/types?
{% endcapture -%}
{% for group in robots.default_groups %}
{{- group.user_agent -}}
{% for rule in group.rules %}
{{- rule -}}
{% endfor %}
{%- if group.user_agent.value == '*' -%}
{{- extra_rules -}}
{%- endif -%}
{%- if group.sitemap != blank -%}
{{ group.sitemap }}
{%- endif -%}
{% endfor %}
To install: Online Store → Themes → … → Edit code. In the Templates folder choose Add a new template, pick robots.txt, replace its contents with this code and save. Check the result at yourstore.com/robots.txt.
robots.txt stops crawling, not indexing. A blocked URL can still appear in Google if other sites link to it. Use a noindex tag for pages that must stay out of search.
How Shopify's robots.txt.liquid works
Shopify writes a sensible robots.txt for every store and updates it over time. To change it, you add a templates/robots.txt.liquid template. The safe way is to loop over robots.default_groups so Shopify's rules keep coming through, then add or remove specific lines. That's what this generator writes.
- Extra rules go inside the
User-agent: *group, right after Shopify's own rules. - Removed rules are skipped by matching their directive and path, for example
Disallow /policies/. - AI crawler blocks are separate groups added after Shopify's groups.
What each option adds
| Option | Lines added |
|---|---|
| Internal search | Disallow: /search and Disallow: /*/search? |
| Single-filter URLs | Disallow: /collections/*?*filter. (plus the translated version) |
| Vendor and type pages | Disallow: /collections/vendors? and /collections/types? |
| Blog tag pages | Disallow: /blogs/*/tagged/ |
| AI training crawlers | GPTBot, ClaudeBot, Google-Extended, Applebot-Extended, CCBot, meta-externalagent |
| AI search crawlers | OAI-SearchBot, Claude-SearchBot, PerplexityBot |
Training and search crawlers are separate on purpose. Blocking Google-Extended or Applebot-Extended doesn't affect Google or Apple search, according to both companies. Blocking the AI search crawlers can keep your products out of ChatGPT search, Claude and Perplexity answers.
Installing and checking
- 01
Add the template
Go to Online Store → Themes → … → Edit code. In Templates, choose Add a new template, pick robots.txt, replace the contents and save.
- 02
Check the live file
Open
yourstore.com/robots.txtand confirm Shopify's rules are still there, followed by yours. - 03
Test important URLs
Use Search Console's URL Inspection on a product and a collection to make sure they're still allowed.
robots.txt controls crawling, not indexing. A blocked URL can still appear in search if it's linked from elsewhere. Don't block ?variant= product URLs: Google Shopping uses them as landing pages.
Questions people ask
Can I edit robots.txt on Shopify?
Yes, by adding a robots.txt.liquid template to your theme. Shopify recommends building on robots.default_groups rather than replacing the file with fixed text.
Does Shopify already block search and filter pages?
Shopify's defaults change over time. Recent versions block sort and multi-filter URLs, but not every store's defaults include /search. Check yours at /robots.txt before adding rules; duplicate lines do no harm.
Should I block GPTBot?
It depends on whether you want your content used to train OpenAI's models. Blocking GPTBot doesn't stop OAI-SearchBot, which is what surfaces sites in ChatGPT search.
Will user-triggered AI fetchers obey this file?
Not always. OpenAI and Perplexity say their user-initiated fetchers may not follow robots.txt, because a person asked for the page.
Related tools
Want this builtinto your store?
I'm Emmanuel, a Shopify and WooCommerce developer. If this tool pointed at something you'd rather have fixed for you (a faster store, cleaner product data, a migration, a custom feature), send me a note.
Emmanuel Asika
Shopify & WooCommerce developer, Dublin
I build and fix online stores: new builds, migrations, speed, SEO and the custom bits apps can't do.
Last reviewed 30 September 2026.
Sources: Shopify: Customize robots.txt · Shopify: robots.txt.liquid template · OpenAI: Overview of OpenAI crawlers · Anthropic: Crawler information · Google: Common crawlers (Google-Extended) · Apple: About Applebot · Common Crawl: CCBot · Meta: Web crawlers · Perplexity: Perplexity crawlers