Start free
← Free SEO tools

AI crawler checker

Is your site blocking
ChatGPT?

Find out in seconds whether your robots.txt lets the 16 crawlers behind ChatGPT, Claude, Perplexity, Gemini and other AI engines read your site, and whether you publish an llms.txt.

Free. We read robots.txt and llms.txt only.

How do big sites set this? We checked 99 popular sites (October 2026): per-bot numbers and a downloadable CSV.

Reference: AI crawler user agents, with what each bot is used for and the vendor's own documentation.

Free, no account. We read your public robots.txt and llms.txt, nothing else.

How to use it

  1. 1
    Type your website

    We read its robots.txt the way the crawlers do, plus its llms.txt.

  2. 2
    Read the verdict per bot

    Search and assistant bots (OAI-SearchBot, Claude-SearchBot, PerplexityBot) decide whether AI answers can cite you. Training bots (GPTBot, ClaudeBot, Google-Extended) only decide whether your pages are used to train models.

  3. 3
    Fix what you did not mean to block

    Blocking training bots is a fair choice. Blocking the search bots usually is not: it takes you out of ChatGPT search, Claude and Perplexity answers.

Questions

Which AI crawlers does it check?

OpenAI (OAI-SearchBot, ChatGPT-User, GPTBot), Anthropic (Claude-SearchBot, Claude-User, ClaudeBot), Perplexity (PerplexityBot, Perplexity-User), Google (Googlebot, Google-Extended), Microsoft (Bingbot, which feeds Copilot), Apple (Applebot-Extended), Meta (Meta-ExternalAgent), Common Crawl (CCBot), ByteDance (Bytespider) and Amazon (Amazonbot).

Should I block GPTBot?

It depends on what you want. GPTBot collects pages to train OpenAI models; blocking it does not remove you from ChatGPT search, which uses OAI-SearchBot. Many publishers block training bots and allow search bots, so their pages can still be cited.

What is llms.txt?

A proposed file at yoursite.com/llms.txt that gives AI tools a short, plain-text map of your site: what it is and which pages matter. It is not a standard yet and no engine promises to use it, but it is cheap to add.

My robots.txt allows everything. Why am I still not cited?

Two common reasons. A CDN or firewall (for example Cloudflare’s setting to block AI bots) can turn crawlers away before robots.txt matters; this check cannot see that. And being allowed is not being cited: AI engines cite pages that answer the question clearly. Our free AI visibility checker shows whether you are cited.

How do you read robots.txt?

Following RFC 9309 and Google’s parser: the group that names the bot wins over the * group, the longest matching rule wins, Allow wins a tie, and * and $ work as wildcards. We check the homepage path (/). A missing robots.txt (404) means everything is allowed.

How common is it to block AI crawlers?

We read the robots.txt of 99 popular sites in October 2026: of the 93 we could read, 26 (28%) blocked GPTBot, 13 (14%) blocked OAI-SearchBot and 55 blocked none of the 16 bots. The full table and CSV are in our study of 99 popular sites. It is a snapshot of what robots.txt says, not of what each site intends.

Is it free?

Yes, with a fair limit per visitor. No account needed.

More free SEO and GEO tools

AI visibility checkerGEO

Does ChatGPT search or Google AI Overviews cite your site? Who do they cite instead?

 Robots.txt and sitemap checkerSEO

Can Google and Bing crawl your site? Is your sitemap valid and listed in robots.txt?

 Robots.txt generatorGEO

Build a robots.txt with your sitemap and blocked paths, plus presets to block AI training bots and keep AI search.

 llms.txt generatorGEO

Create an llms.txt in the llmstxt.org format. A proposal, not a ranking tactic, and the page says so.

 Schema markup generatorSEO

JSON-LD for FAQPage, Article, LocalBusiness and Organization, with required-field hints. No fake review schema.

 Open Graph tag generatorSEO

Write og: and Twitter card tags, preview the share card and catch long titles and non-https images.

 Blog post ideasContent

Real searches your site could write a post for, with approximate monthly volume and a suggested post format.

 SERP snippet checkerSEO

Preview your title and meta description in Google, with the pixel widths Google truncates at.

 Keyword combinerSEO

Combine up to three keyword lists into every long-tail variation, ready to copy.

 Keyword density checkerContent

Paste an article: see the words and phrases it repeats most, and spot keyword stuffing.

 Word counter and reading timeContent

Live word, character, sentence and paragraph counts, reading time, and rough length guides for titles, meta descriptions and posts.

 Readability checkerContent

Flesch reading ease, grade level, sentence length, long sentences and a passive-voice hint for English text. Syllables are estimated.

 UTM builderSEO

Add source, medium and campaign to any link so analytics shows where each visit came from. Copy the tagged URL.

 Title case converterContent

Switch any headline or paragraph to title case, sentence case, UPPERCASE or lowercase. Acronyms and brand names you typed are kept.

 hreflang generatorSEO

List your language versions and copy the full set of hreflang link tags, with an optional x-default.

 Heading structure checkerContent

Paste HTML or markdown and see the H1 to H6 outline, with missing or multiple H1s, skipped levels, empty and long headings flagged as guidance.

 Robots meta tag generatorSEO

Build a robots meta tag and X-Robots-Tag header with noindex, nosnippet and max-snippet, plus Apache and nginx snippets. Directives documented by Google.

 Image alt text checkerContent

Paste HTML and list every image with its alt text. Flags missing alt, empty alt to double-check, file names, long text and filler phrases as guidance.

 Keyword grouperSEO

Paste up to 2,000 keywords and group them by shared words into a first draft of topic clusters. Groups by words, not search intent. Copy or download CSV.

 Link and anchor text checkerContent

Paste HTML and list every link with its anchor text and rel values. Flags generic and empty anchors, placeholder hrefs and mixed anchors as guidance.

 FAQ schema generatorSEO

Paste your questions and answers and copy valid FAQPage JSON-LD, safely escaped. Expect valid markup, not a guaranteed rich result.

 Sitemap generatorSEO

Paste a list of URLs and download a valid sitemap.xml: duplicates removed, one site, XML-escaped, 50,000-URL warning.

 SEO checklist for blog postsContent

An interactive pre-publish checklist: intent, on-page, technical and after-publish checks, saved in your browser, with a free tool linked where one helps.

 Content calendar generatorContent

Turn a list of topics into a dated publishing schedule. Pick the start date, weekdays and pace; copy it or download a CSV.

 URL slug generatorSEO

Turn any title into a short, clean, SEO-friendly URL slug, in any language.

 SEO content cost calculatorContent

Cost per blog post and per month: freelancer, agency, in-house, AI tools or Balzac. Your numbers, editable.

Let Balzac write the articles.

Type your website. Balzac finds the searches you can win, writes articles built to rank on Google and get cited by ChatGPT, and publishes them for you.

3 free articles · No card needed