Robots.txt and sitemap checker
Can Google crawl
your site?
Check in seconds whether your robots.txt lets Google and Bing crawl your site, and whether your sitemap loads, is valid and is listed where crawlers look.
| Crawler | Homepage |
|---|
Sitemaps
| Sitemap | Status |
|---|
Free, no account. We read your public robots.txt and sitemap files, nothing else.
How to use it
- 1Type your website
We fetch its robots.txt and the sitemap files it points to (or /sitemap.xml if it names none).
- 2Read the verdicts
Googlebot and Bingbot: allowed or blocked for your homepage, and the rule that decided it. Then each sitemap: does it load, is it valid, how many URLs does it list.
- 3Fix the red and amber lines first
A blocked homepage or a broken sitemap are the ones that cost you. Add a Sitemap: line to robots.txt if it is missing, then submit the sitemap in Google Search Console and Bing Webmaster Tools.
Questions
How do I check my robots.txt?
Type your domain above. We fetch yoursite.com/robots.txt, read it the way Google does and tell you whether Googlebot and Bingbot may crawl your homepage, and which rule decides it. You can also open yoursite.com/robots.txt in a browser to read the raw file.
How do I know if my sitemap is valid?
We check that the sitemap URL loads and that it is a real sitemap: an XML file with a <urlset> (a list of pages) or a <sitemapindex> (a list of other sitemaps). We then count the entries. For a sitemap index we count the child sitemaps but do not open them.
Where should the sitemap be listed?
Add a line like Sitemap: https://yoursite.com/sitemap.xml to robots.txt. It is not required, but it is the standard way for crawlers to find it. If robots.txt names no sitemap, we try /sitemap.xml.
What does it not check?
It reads robots.txt rules and sitemap files, nothing more. It does not check noindex tags or headers, canonical tags, crawl budget, page speed, or whether Google has actually indexed your pages. Search Console shows that. It also only tests the homepage path (/), not every URL.
My robots.txt allows Google. Why are my pages not indexed?
Allowed to crawl is not the same as indexed. Pages can carry a noindex tag, be duplicates or thin, be hard to reach by links, or simply not have been picked up yet. The URL Inspection tool in Google Search Console tells you the reason for a specific page.
Does blocking a page in robots.txt remove it from Google?
Not reliably. robots.txt controls crawling, not indexing: a blocked URL can still appear in results if other sites link to it. To keep a page out of results, let it be crawled and use a noindex tag.
Which AI bots can read my site?
That is a different check. Our free AI crawler checker tests GPTBot, OAI-SearchBot, ClaudeBot, PerplexityBot, Google-Extended and more against the same robots.txt.
How many popular sites list a sitemap in robots.txt?
In our check of 99 popular sites on 2026-10-06, 71 of the 92 readable robots.txt files listed at least one sitemap, and 64 of those 71 pointed to a sitemap that loaded. Method and data are in the study.
Is it free?
Yes, with a fair limit per visitor. No account needed.
More free SEO and GEO tools
Does ChatGPT search or Google AI Overviews cite your site? Who do they cite instead?
AI crawler checkerGEOIs your robots.txt blocking ChatGPT, Claude, Perplexity or Gemini? 16 AI bots checked, plus llms.txt.
Robots.txt generatorGEOBuild a robots.txt with your sitemap and blocked paths, plus presets to block AI training bots and keep AI search.
llms.txt generatorGEOCreate an llms.txt in the llmstxt.org format. A proposal, not a ranking tactic, and the page says so.
Schema markup generatorSEOJSON-LD for FAQPage, Article, LocalBusiness and Organization, with required-field hints. No fake review schema.
Open Graph tag generatorSEOWrite og: and Twitter card tags, preview the share card and catch long titles and non-https images.
Blog post ideasContentReal searches your site could write a post for, with approximate monthly volume and a suggested post format.
SERP snippet checkerSEOPreview your title and meta description in Google, with the pixel widths Google truncates at.
Keyword combinerSEOCombine up to three keyword lists into every long-tail variation, ready to copy.
Keyword density checkerContentPaste an article: see the words and phrases it repeats most, and spot keyword stuffing.
Word counter and reading timeContentLive word, character, sentence and paragraph counts, reading time, and rough length guides for titles, meta descriptions and posts.
Readability checkerContentFlesch reading ease, grade level, sentence length, long sentences and a passive-voice hint for English text. Syllables are estimated.
UTM builderSEOAdd source, medium and campaign to any link so analytics shows where each visit came from. Copy the tagged URL.
Title case converterContentSwitch any headline or paragraph to title case, sentence case, UPPERCASE or lowercase. Acronyms and brand names you typed are kept.
hreflang generatorSEOList your language versions and copy the full set of hreflang link tags, with an optional x-default.
Heading structure checkerContentPaste HTML or markdown and see the H1 to H6 outline, with missing or multiple H1s, skipped levels, empty and long headings flagged as guidance.
Robots meta tag generatorSEOBuild a robots meta tag and X-Robots-Tag header with noindex, nosnippet and max-snippet, plus Apache and nginx snippets. Directives documented by Google.
Image alt text checkerContentPaste HTML and list every image with its alt text. Flags missing alt, empty alt to double-check, file names, long text and filler phrases as guidance.
Keyword grouperSEOPaste up to 2,000 keywords and group them by shared words into a first draft of topic clusters. Groups by words, not search intent. Copy or download CSV.
Link and anchor text checkerContentPaste HTML and list every link with its anchor text and rel values. Flags generic and empty anchors, placeholder hrefs and mixed anchors as guidance.
FAQ schema generatorSEOPaste your questions and answers and copy valid FAQPage JSON-LD, safely escaped. Expect valid markup, not a guaranteed rich result.
Sitemap generatorSEOPaste a list of URLs and download a valid sitemap.xml: duplicates removed, one site, XML-escaped, 50,000-URL warning.
SEO checklist for blog postsContentAn interactive pre-publish checklist: intent, on-page, technical and after-publish checks, saved in your browser, with a free tool linked where one helps.
Content calendar generatorContentTurn a list of topics into a dated publishing schedule. Pick the start date, weekdays and pace; copy it or download a CSV.
URL slug generatorSEOTurn any title into a short, clean, SEO-friendly URL slug, in any language.
SEO content cost calculatorContentCost per blog post and per month: freelancer, agency, in-house, AI tools or Balzac. Your numbers, editable.
Let Balzac write the articles.
Type your website. Balzac finds the searches you can win, writes articles built to rank on Google and get cited by ChatGPT, and publishes them for you.
3 free articles · No card needed