AI Search Visibility / Crawler Access
Is GPTBot, ClaudeBot or PerplexityBot Blocked From Your Site?
Many sites silently block the exact crawlers that feed AI answer engines — usually by accident, via a default robots.txt template or a dev-time rule that never got reverted. One narrow, verifiable check: paste your URL and see which of the 10 named AI crawlers can actually read your site.
The Honest Frame
Blocked ≠ invisible to that AI, in every case
Access is necessary but not sufficient, and the reverse isn't a clean rule either. GPTBot feeds OpenAI model training plus some retrieval — blocking it does not switch off every way ChatGPT might reference your site, since browsing/search fetches can use a different user-agent some sites don't think to block. CCBot seeds Common Crawl, which several LLMs train on indirectly, so its effect is diffuse and delayed rather than immediate. This tool reports the one thing that IS a clean, verifiable fact — what your robots.txt actually says about each crawler, quoting the exact line — and explains what that specific rule does and doesn't control, instead of collapsing it to a single misleading verdict.
See the difference
What an accidental AI-crawler block looks like in robots.txt, why it happens, and the one-line fix.
02·Practical use cases
Who needs to check AI-crawler access
Site owners migrating hosts or CDNs
Confirm a new hosting panel, WAF, or CDN didn't ship a default robots.txt that blocks AI crawlers by name.
A line-by-line answer for each of 10 named crawlers, instead of guessing from a hosting provider's changelog.
Teams that fought a bot-traffic spike
Check whether an emergency 'block all scrapers' rule swept up PerplexityBot, ClaudeBot or GPTBot along with the abusive traffic.
Confirmation of exactly which legitimate AI crawlers got caught in a defensive rule that was written in a hurry.
Anyone who launched from a staging environment
Verify a blanket 'Disallow: /' used during development was actually removed before the site went live, not just forgotten.
A clean, dated answer instead of trusting memory about whether the dev-only rule was reverted.
Content and SEO teams tracking AI visibility
Add crawler access as a standing item in a pre-publish or quarterly technical check, the same way they already check indexability.
A fast, free, repeatable check that quotes the exact rule so a developer can act on it without re-deriving the robots.txt logic themselves.
03·How it works
How the crawler check works
No signup, no email, no install — the bot fetches your site's robots.txt and the target page directly over Telegram.
GPTBot, OAI-SearchBot, ChatGPT-User, PerplexityBot, Perplexity-User, ClaudeBot, Google-Extended, CCBot, Applebot-Extended and Amazonbot are matched against the parsed robots.txt rules one at a time — never collapsed into a single pass/fail score.
If a crawler is blocked, the report shows the literal robots.txt line responsible (for example `Disallow: /`) and which user-agent group it fell under, so you can verify it yourself in seconds.
Beyond robots.txt, the bot also reads the page's meta-robots tag for a stray `noindex`/`nofollow` that would undercut the fix even after robots.txt is corrected.
04·Same tool · in Telegram
Telegram
Check your robots.txt right now
@vustSEObot · Paste your URL in @vustSEObot — it checks 10 named AI crawlers in one pass and quotes the exact blocking rule, if any. Free, 10 checks/day, no signup.
05·Quality & trust
Honest about what this check does and doesn't tell you
Every finding is a quoted, verifiable fact
The report never says 'this looks blocked' — it quotes the exact robots.txt line and the user-agent group that matched, so you can open the file yourself and confirm it in ten seconds. That's the entire trust anchor of this check.
Blocked ≠ invisible to that AI product
GPTBot governs OpenAI model training and some retrieval; it is not the single switch for every way ChatGPT might reference a page, since live browsing/search fetches can use a different crawler some sites never think to block. This tool tells you what the rule says, not what every downstream AI product does with that signal.
Allowed ≠ guaranteed citation
Letting every crawler read your site is necessary, not sufficient — an allowed crawler still has to find the page, parse it, and judge it worth citing. This check answers the access question honestly and stops there; no tool can promise the citation on top of it.
Free, fast, and narrow on purpose
This is one Tier-1 mechanical check, not the whole GEO picture. For structured data, answer-block presence, chunkability and freshness in the same pass, the full audit at @vustSEObot covers all of it — this page exists because the crawler-access finding is common, silent, and worth checking on its own.
Frequently asked questions
More of the GEO/AEO audit
AI Search Visibility (GEO/AEO)
The general framework — why AI cites some sources and not yours.
Full GEO/AEO Audit
The complete Tier-1 mechanical audit beyond just crawler access.
FAQ Schema Generator
Turn your content into ready-to-paste FAQPage JSON-LD.
AI Chat (all models)
Ask a model to help you read or edit a robots.txt file.
Ready when you are
Find out if AI answer engines can even read your site.
GPTBot, OAI-SearchBot, ChatGPT-User, PerplexityBot, Perplexity-User, ClaudeBot, Google-Extended, CCBot, Applebot-Extended, Amazonbot — checked individually, with the exact robots.txt line quoted back. Free, in Telegram, no signup.