Robots.txt tester
Allowed
or blocked?
Paste a robots.txt, pick a crawler and list some URLs. See which are allowed or blocked and exactly which rule decided it.
What Google says about matching
"Only one group is valid for a particular crawler."
"When matching robots.txt rules to URLs, crawlers use the most specific rule based on the length of the rule path. In case of conflicting rules, including those with wildcards, Google uses the least restrictive rule."
"designates 0 or more instances of any valid character." (about *) and "designates the end of the URL." (about $)
"The field name (disallow) is case-insensitive, but its value is case-sensitive."
Free, no account, nothing leaves your browser.
How to use it
- 1Paste your robots.txt
Open yoursite.com/robots.txt in a browser, copy the text and paste it in. Or draft a new version here before you publish it.
- 2Add a user agent and the URLs to test
Use a crawler name such as Googlebot or Googlebot-News, and one URL or path per line. Full URLs and paths that start with / both work.
- 3Read the verdict and the rule behind it
Each URL shows allowed or blocked, the rule that decided it with its line number, and every other rule that matched. Test your fix before you ship it.
Questions
Which matching rules does it use?
The ones Google documents on its robots.txt page: the most specific rule, by the length of its path, wins; when an allow and a disallow rule tie, the allow rule wins; * matches any run of characters; $ at the end of a rule means the end of the URL; paths are case-sensitive. The page quotes Google's own wording above the tool.
How does it choose the group for my user agent?
It looks for the group whose user-agent value is the same as the crawler name you typed, ignoring case, and falls back to the * group when there is none. Several groups for the same name are merged, and a specific group is never combined with the * group, as Google describes. It compares the product token only, so a full browser-style user agent string is cut down to its first token.
Is this exactly what Googlebot does?
No. It is a reimplementation of the documented rules, not Google's own parser. It tests only the path and query of each URL, and ignores the host, the protocol and the port. It does not model how Google handles a robots.txt that cannot be fetched, the 500 KiB size limit, or crawlers that need special handling. For an authoritative answer on a live page, use the URL Inspection tool in Search Console.
Does blocked here mean removed from Google?
No. robots.txt controls crawling, not indexing: Google documents that it can still index a disallowed URL and show it without a snippet. To keep a page out of results, use a noindex rule on a page that can be crawled.
Is my robots.txt uploaded?
No. Everything runs as JavaScript in your browser and nothing is sent to a server.
What happens with an empty Disallow or Allow line?
Google documents that a disallow rule without a path is ignored and that an allow rule with no path is ignored. This tool ignores them too, so an empty Disallow: does not block anything.
More free SEO and GEO tools
Does ChatGPT search or Google AI Overviews cite your site? Who do they cite instead?
AI crawler checkerGEOIs your robots.txt blocking ChatGPT, Claude, Perplexity or Gemini? 16 AI bots checked, plus llms.txt.
Robots.txt and sitemap checkerSEOCan Google and Bing crawl your site? Is your sitemap valid and listed in robots.txt?
Robots.txt generatorGEOBuild a robots.txt with your sitemap and blocked paths, plus presets to block AI training bots and keep AI search.
llms.txt generatorGEOCreate an llms.txt in the llmstxt.org format. A community proposal; Google Search ignores it, and the page says so.
Schema markup generatorSEOJSON-LD for FAQPage, Article, LocalBusiness and Organization, with required-field hints. No fake review schema.
Open Graph tag generatorSEOWrite og: and Twitter card tags, preview the share card and catch long titles and non-https images.
Blog post ideasContentReal searches your site could write a post for, with approximate monthly volume and a suggested post format.
SERP snippet checkerSEOPreview your title and meta description in Google, with the pixel widths Google truncates at.
Meta description length checkerSEOCount characters and estimate pixel width for a meta description. Google sets no character limit, so the pixel line is a labelled approximation.
Keyword combinerSEOCombine up to three keyword lists into every long-tail variation, ready to copy.
Keyword density checkerContentPaste an article: see the words and phrases it repeats most, and spot keyword stuffing.
Word counter and reading timeContentLive word, character, sentence and paragraph counts, reading time, and rough length guides for titles, meta descriptions and posts.
Readability checkerContentFlesch reading ease, grade level, sentence length, long sentences and a passive-voice hint for English text. Syllables are estimated.
UTM builderSEOAdd source, medium and campaign to any link so analytics shows where each visit came from. Copy the tagged URL.
Title case converterContentSwitch any headline or paragraph to title case, sentence case, UPPERCASE or lowercase. Acronyms and brand names you typed are kept.
hreflang generatorSEOList your language versions and copy the full set of hreflang link tags, with an optional x-default.
Heading structure checkerContentPaste HTML or markdown and see the H1 to H6 outline, with missing or multiple H1s, skipped levels, empty and long headings flagged as guidance.
Robots meta tag generatorSEOBuild a robots meta tag and X-Robots-Tag header with noindex, nosnippet and max-snippet, plus Apache and nginx snippets. Directives documented by Google.
Image alt text checkerContentPaste HTML and list every image with its alt text. Flags missing alt, empty alt to double-check, file names, long text and filler phrases as guidance.
Keyword grouperSEOPaste up to 2,000 keywords and group them by shared words into a first draft of topic clusters. Groups by words, not search intent. Copy or download CSV.
Link and anchor text checkerContentPaste HTML and list every link with its anchor text and rel values. Flags generic and empty anchors, placeholder hrefs and mixed anchors as guidance.
SEO content brief generatorContentTurn a keyword, intent and reference URLs into a brief template: title options, outline, questions, on-page checklist and E-E-A-T prompts. Copy or download markdown.
301 redirect generatorSEOPaste old and new URLs and get redirect rules for Apache, nginx, Netlify and Cloudflare _redirects and Vercel. Validates input and flags duplicates, chains and loops.
IndexNow generatorSEOMake an IndexNow key, the key file and GET, POST and curl requests for your changed URLs. Builds the request only; Google is not a listed participant.
Sitemap URL extractorSEOPaste sitemap.xml and list every URL and lastmod. Flags more than 50,000 entries, invalid dates, duplicates, non-http(s) URLs and mixed hosts. Copy or CSV.
Canonical tag checkerSEOPaste page HTML and check its rel=canonical: head vs body, multiple tags, relative URLs, fragments, http/https and www mismatches, noindex and og:url.
FAQ schema generatorSEOPaste your questions and answers and copy valid FAQPage JSON-LD, safely escaped. Valid markup only: Google no longer shows FAQ rich results.
Article schema generatorSEOArticle, BlogPosting or NewsArticle JSON-LD with several images and authors, date checks and a checklist of what Google documents as recommended. Parsed back before you copy.
Breadcrumb schema generatorSEOType your breadcrumb trail and copy BreadcrumbList JSON-LD with numbered positions and the last-item rule from Google's docs. Parsed back before you copy.
JSON-LD checkerSEOPaste page HTML or a snippet and list every ld+json block: JSON syntax errors, @context, @type and @graph items. Parsed in your browser.
Keyword cannibalization finderSEOPaste a Search Console export with Query and Page columns and list queries where two or more of your pages got impressions, with clicks and position per page. CSV export, in your browser.
Striking distance keyword finderSEOPaste a Search Console export and list the queries ranking just off the top of page one, positions 8 to 20 by default, sorted by impressions. CSV export, in your browser.
Content decay finderSEOPaste two Search Console exports, an earlier period and a recent one, and list the pages or queries that lost the most clicks. CSV export, in your browser.
Question query finderContentPaste a Search Console export and list the queries phrased as questions, grouped by question word and sorted by impressions. CSV export, in your browser.
Low CTR finderSEOPaste a Search Console export and list the queries or pages that rank high but earn few clicks, with your own position, impressions and CTR limits. CSV export, in your browser.
CTR by position bucketSEOPaste a Search Console export and see impressions, clicks and CTR grouped by average position: 1, 2, 3, 4-5, 6-10, 11-20 and 21+. CSV export, in your browser.
New vs lost queriesSEOPaste two Search Console exports and list the queries that appear only in the recent period and the ones only in the earlier one, with click and impression changes. CSV export, in your browser.
Branded vs non-branded queriesSEOPaste a Search Console queries export and your brand terms and see queries, clicks, impressions, CTR and position for branded and non-branded searches. CSV export, in your browser.
llms.txt validatorGEOPaste an llms.txt and check it against the llmstxt.org proposal: H1, summary, H2 link lists, Optional section, link format. A community proposal, not an official standard. In your browser.
XML sitemap validatorSEOPaste sitemap XML and check well-formedness, urlset or sitemapindex, absolute loc URLs, the 50,000 URL and 50MB limits, lastmod format, duplicates and mixed hosts. In your browser.
Internal link opportunity finderContentPaste your page URLs and titles and an article, and see which pages share words with it, with the phrase to link. Plain word matching, not a relevance score. In your browser.
hreflang validatorSEOPaste the head HTML of each language version, or a sitemap with xhtml:link entries, and check codes, absolute URLs, self-reference and return links. Based on Google's documented rules. In your browser.
URL structure checkerSEOPaste a list of URLs and flag upper case, underscores, unencoded characters, fragments used as routes, odd or many parameters and ID-number segments, against what Google documents. In your browser.
HTTP header checkerSEOPaste response headers (curl -sIL) and check the status, redirect chain, X-Robots-Tag directives and Link header canonical and hreflang values, against what Google documents. In your browser.
Structured data property checkerSEOPaste page HTML or JSON-LD and check Product, BreadcrumbList, LocalBusiness, Article, Organization and review markup for the required and recommended properties Google documents. In your browser.
Redirect map checkerSEOPaste old URL to new URL pairs and find loops, chains, self redirects and conflicting sources, then copy a flattened map that skips the middle hops. In your browser.
Duplicate title and meta description finderSEOPaste a crawl export of URL, title and description and find pages that share a title or description, plus missing ones. Exact matches only. In your browser.
Sitemap diff checkerSEOPaste an old and a new sitemap and see which URLs were added, which were removed and which lastmod values changed. Exact URL matching. In your browser.
Robots.txt diff checkerSEOPaste two robots.txt versions and see the rules added or removed under each user agent, changed sitemap lines, and a warning if the new file blocks the whole site. In your browser.
Googlebot log analyzerSEOPaste access-log lines and see the hits claiming to be Googlebot by status code and day, with the URLs that returned redirects, 404s and 5xx. Claimed, not verified. In your browser.
Table of contents generatorContentPaste HTML or Markdown and get an anchor-linked table of contents with slugged heading ids, as HTML or Markdown. Copy it, in your browser.
Meta tag checkerSEOPaste page HTML and list its title, meta description, viewport, charset, lang, canonical, robots, Open Graph and Twitter tags, with notes from Google's docs.
QR code generator with UTMSEOTag a URL with UTM source, medium and campaign and get its QR code, drawn in your browser. Download PNG or SVG. A static code: scans show up as tagged visits in analytics.
Sitemap generatorSEOPaste a list of URLs and download a valid sitemap.xml: duplicates removed, one site, XML-escaped, 50,000-URL warning.
SEO checklist for blog postsContentAn interactive pre-publish checklist: intent, on-page, technical and after-publish checks, saved in your browser, with a free tool linked where one helps.
Content calendar generatorContentTurn a list of topics into a dated publishing schedule. Pick the start date, weekdays and pace; copy it or download a CSV.
URL slug generatorSEOTurn any title into a short, clean, SEO-friendly URL slug, in any language.
SEO content cost calculatorContentCost per blog post and per month: freelancer, agency, in-house, AI tools or Balzac. Your numbers, editable.
Let Balzac write the articles.
Type your website. Balzac finds the searches you can win, writes articles built to rank on Google and get cited by ChatGPT, and publishes them for you.
3 free articles · No card needed