EStore Tools

AI Crawler Checker

See exactly which AI bots your robots.txt lets reach a page, and which it blocks.

Describe the result you need. AI will set up the tool and ask for any missing details.

Configure with AI

Enter to submit · Shift + Enter for a new line

Sent to AI. Leave out passwords and private customer data.

What this shows

Paste any page on your store and this fetches your robots.txt and works out, bot by bot, whether that specific path is allowed or blocked, the same way each crawler would evaluate it: the most specific matching rule for that bot's own group (or the wildcard * group if it has none) wins.

Results are grouped into search and answer bots (the ones that can send an AI product's users to your page), training bots (the ones that only collect data for future models), and Googlebot as a reference point, since it's governed by the same file but serves Google Search, not an AI product.

Questions people ask

A bot shows as blocked but I never added a rule for it. Why?

It's likely caught by your wildcard User-agent: * group. Add a specific User-agent: group for that bot with its own Allow rule to carve out an exception.

Can I fix problems it finds right here?

Use the AI Crawler robots.txt Generator to build corrected rules, then paste them into your robots.txt.

It says a bot may ignore robots.txt anyway. What's the point of checking it then?

It's still the documented way to signal your preference, and most crawlers honor it. The note flags the few where the company has said a live, user-triggered fetch might not check the file, so you know not to rely on it there.

Does this check work on non-Shopify, non-WooCommerce sites?

Yes. It only reads your robots.txt and doesn't assume any platform.

Related tools

AI Crawler robots.txt GeneratorChoose exactly which AI bots can train on or search your store, bot by bot.AI Crawler Render CheckerSee what your page's raw HTML contains, before any JavaScript runs, the way most AI bots see it.AI Agent Readiness CheckerSee what AI shopping agents can find on your store: feeds, structured data, crawler access and agent protocols.GA4 AI Traffic Regex GeneratorBuild the regex that groups ChatGPT, Perplexity, Gemini and other AI assistants into their own GA4 channel.

Want this builtinto your store?

I'm Emmanuel, a Shopify and WooCommerce developer. If this tool pointed at something you'd rather have fixed for you (a faster store, cleaner product data, a migration, a custom feature), send me a note.

Emmanuel Asika

Emmanuel Asika

Shopify & WooCommerce developer, Dublin

I build and fix online stores: new builds, migrations, speed, SEO and the custom bits apps can't do.

hello@emmanuelasika.com
See my work

Last reviewed 30 September 2026.