SourceContact
Free · No sign-up · Run from an overseas check point

AI Crawler Access CheckThe crawlers behind six AI platforms — can they reach your site?

We read your robots.txt and judge, one by one, whether the crawlers behind ChatGPT, Claude, Perplexity, Gemini, Microsoft Copilot and Google AI Overviews are allowed in. Each tier is judged separately: training, retrieval and user-triggered bots. Blocking one of them has a very different consequence from blocking another, and reading them as one number gives you the opposite answer.

This checks whether they can reach you. It does not measure how often your brand is mentioned in AI answers — that is what continuous monitoring does. See continuous monitoring →

Choose your business type so the report uses the right buyer perspective

No account, no email, no phone number. Each result gets a permanent link you can share.

Blocking GPTBot does not remove you from ChatGPT's answers

This is the one that gets inverted most often. OpenAI, Anthropic and Perplexity each run three tiers of crawler, and they do different jobs:

Training bots (GPTBot, ClaudeBot, CCBot, Amazonbot, Applebot-Extended) collect training data. Blocking them is a copyright decision and has nothing to do with whether you can be cited — so we never score this tier, we only report its state.

Retrieval bots (OAI-SearchBot, Claude-SearchBot, PerplexityBot, Bingbot, Googlebot) build the index behind AI answers. Blocking any one of them means disappearing from that AI's answers, so a site-wide block here is always a failure.

User-triggered bots (ChatGPT-User, Claude-User, Perplexity-User) fetch in real time when someone asks. Blocking them affects that one conversation, so it is a warning.

Treating all three as one thing is a mistake shared by a great deal of GEO writing. A report that turns "GPTBot is blocked" into "you are invisible in ChatGPT" sends you off to fix something that does not change the outcome.

Google AI Overviews are governed by Googlebot, not Google-Extended

Google is the easiest one to get wrong. Intuition says the crawler with "Extended" in its name is the AI one. The official documentation says the opposite.

In AI Features and Your Website, Google writes that AI is built into Search and integral to how Search functions, which is why robots.txt directives for Googlebot are the control site owners use to manage how their sites are crawled for Search. AI Overviews and AI Mode are grounded in the Search index, so blocking Googlebot means leaving AI Overviews too.

Google-Extended, meanwhile, is documented as not impacting a site's inclusion in Google Search and not being used as a ranking signal. It governs training and grounding for Gemini apps and Vertex AI.

Both are in our table, on different tiers: Googlebot is a retrieval bot and a block is a failure; a Google-Extended block is a warning.

Which lines in that robots.txt did you actually write?

The check also identifies CDN-injected managed blocks. Some CDNs insert a whole section of AI-crawler rules into robots.txt without the site owner asking for it.

This item is always advisory and never scored. Its only job is to let the findings above be attributed correctly: when you see "ClaudeBot is blocked", the first thing you need to know is whether you wrote that line — otherwise you go off to change something you never wrote.

Questions

How many crawlers does the check cover?

Fifteen, all of them crawlers of overseas AI platforms and search engines, sorted into training, retrieval and user-triggered tiers. Every tier assignment is backed by the vendor's own documentation, and the judgement logic ships with the engine under AGPL-3.0, so you can check it line by line.

Why is blocking a training bot not scored against me?

Because it has nothing to do with whether you can be cited. Whether to let your content into model training data is a copyright and commercial judgement, and companies legitimately answer it differently. Scoring it would make a non-technical decision for you.

Do you check Chinese AI platforms' crawlers?

Not currently. This version covers overseas platforms only. We do not guess at, or list, crawlers that are not in the table.

Do I get an overall score?

No. The checks differ too much in importance, and once they are compressed into a single number that number cannot guide any action. We give each finding with its evidence and its limits.

How does this relate to your paid monitoring?

This page checks the technical layer: whether crawlers can reach you. Paid continuous monitoring checks the outcome layer: whether your brand actually appears in answers across the six AI platforms. The first is a precondition for the second, but they measure different things.

The judging logic is fully open source. The engine behind this page is published under the GNU AGPL-3.0. How every conclusion is reached, and where its limits lie, is written in the code and its comments and can be checked line by line: github.com/Saqierma/miaowageo