Start free
← Free SEO tools

Robots.txt generator

Your robots.txt,
AI bots included.

Build a robots.txt with your sitemap, blocked paths and a choice for each AI bot: block training crawlers, keep AI search visible, or anything in between.

Search and assistant botsFetch pages so answers can use and link to them.
Training botsCollect content to train models.

Before you publish

    Verify the live file: Robots.txt and sitemap checker · AI crawler checker · llms.txt generator

    Editing robots.txt on your platform: WordPress · Shopify · Wix · Ghost · Framer · HubSpot.

    Free, no account, nothing leaves your browser.

    How to use it

    1. 1
      Pick a preset

      Start from "Allow everything", "Block AI training, allow AI search" or "Block all AI bots". Then flip any single bot to Allow or Block.

    2. 2
      Add your rules

      List the paths to keep crawlers out of (one per line, like /admin/ or /cart/) and your sitemap URL or URLs. Crawl-delay is optional and Google ignores it.

    3. 3
      Copy or download, then verify

      Save the file as robots.txt at the root of your domain (https://example.com/robots.txt). Then run the live file through the robots.txt checker and the AI crawler checker.

    Questions

    What is a robots.txt file?

    A plain text file at the root of your domain that tells crawlers which paths they may fetch. Well-behaved bots read it before crawling. It is a request, not access control.

    Does robots.txt stop a page from being indexed?

    No. It controls crawling, not indexing. A blocked URL can still appear in search results if other pages link to it, just without a description. To keep a page out of the index, let it be crawled and use a noindex meta tag or header, or put it behind a login.

    Which AI bots should I block?

    It depends on what you want. Training bots (GPTBot, ClaudeBot, Google-Extended, Applebot-Extended, Meta-ExternalAgent, CCBot, Bytespider) collect content to train models. Search and assistant bots (OAI-SearchBot, ChatGPT-User, Claude-SearchBot, Claude-User, PerplexityBot, Perplexity-User, Googlebot, Bingbot, Amazonbot) fetch pages so answers can use and link to them. Many sites block the first group and allow the second.

    Does Google respect Crawl-delay?

    No. Googlebot ignores the Crawl-delay directive. Bing and some other crawlers honor it. If a bot is overloading your server, rate limit it at the server or CDN instead.

    Will every AI bot obey my robots.txt?

    Reputable operators say their bots follow it, but it is voluntary. A scraper can ignore it, and a CDN or firewall can block or allow bots regardless of what the file says. Check the live result with the AI crawler checker.

    Why do allowed bots get their own group?

    A bot that has its own User-agent group follows only that group and ignores the "*" group. So allowed bots repeat your disallowed paths and crawl-delay, otherwise naming them would accidentally open the paths you wanted closed.

    Is my input sent anywhere?

    No. The file is built in your browser and nothing is uploaded.

    More free SEO and GEO tools

    AI visibility checkerGEO

    Does ChatGPT search or Google AI Overviews cite your site? Who do they cite instead?

     AI crawler checkerGEO

    Is your robots.txt blocking ChatGPT, Claude, Perplexity or Gemini? 16 AI bots checked, plus llms.txt.

     Robots.txt and sitemap checkerSEO

    Can Google and Bing crawl your site? Is your sitemap valid and listed in robots.txt?

     llms.txt generatorGEO

    Create an llms.txt in the llmstxt.org format. A proposal, not a ranking tactic, and the page says so.

     Schema markup generatorSEO

    JSON-LD for FAQPage, Article, LocalBusiness and Organization, with required-field hints. No fake review schema.

     Open Graph tag generatorSEO

    Write og: and Twitter card tags, preview the share card and catch long titles and non-https images.

     Blog post ideasContent

    Real searches your site could write a post for, with approximate monthly volume and a suggested post format.

     SERP snippet checkerSEO

    Preview your title and meta description in Google, with the pixel widths Google truncates at.

     Keyword combinerSEO

    Combine up to three keyword lists into every long-tail variation, ready to copy.

     Keyword density checkerContent

    Paste an article: see the words and phrases it repeats most, and spot keyword stuffing.

     Word counter and reading timeContent

    Live word, character, sentence and paragraph counts, reading time, and rough length guides for titles, meta descriptions and posts.

     Readability checkerContent

    Flesch reading ease, grade level, sentence length, long sentences and a passive-voice hint for English text. Syllables are estimated.

     UTM builderSEO

    Add source, medium and campaign to any link so analytics shows where each visit came from. Copy the tagged URL.

     Title case converterContent

    Switch any headline or paragraph to title case, sentence case, UPPERCASE or lowercase. Acronyms and brand names you typed are kept.

     hreflang generatorSEO

    List your language versions and copy the full set of hreflang link tags, with an optional x-default.

     Heading structure checkerContent

    Paste HTML or markdown and see the H1 to H6 outline, with missing or multiple H1s, skipped levels, empty and long headings flagged as guidance.

     Robots meta tag generatorSEO

    Build a robots meta tag and X-Robots-Tag header with noindex, nosnippet and max-snippet, plus Apache and nginx snippets. Directives documented by Google.

     Image alt text checkerContent

    Paste HTML and list every image with its alt text. Flags missing alt, empty alt to double-check, file names, long text and filler phrases as guidance.

     Keyword grouperSEO

    Paste up to 2,000 keywords and group them by shared words into a first draft of topic clusters. Groups by words, not search intent. Copy or download CSV.

     Link and anchor text checkerContent

    Paste HTML and list every link with its anchor text and rel values. Flags generic and empty anchors, placeholder hrefs and mixed anchors as guidance.

     FAQ schema generatorSEO

    Paste your questions and answers and copy valid FAQPage JSON-LD, safely escaped. Expect valid markup, not a guaranteed rich result.

     Sitemap generatorSEO

    Paste a list of URLs and download a valid sitemap.xml: duplicates removed, one site, XML-escaped, 50,000-URL warning.

     SEO checklist for blog postsContent

    An interactive pre-publish checklist: intent, on-page, technical and after-publish checks, saved in your browser, with a free tool linked where one helps.

     Content calendar generatorContent

    Turn a list of topics into a dated publishing schedule. Pick the start date, weekdays and pace; copy it or download a CSV.

     URL slug generatorSEO

    Turn any title into a short, clean, SEO-friendly URL slug, in any language.

     SEO content cost calculatorContent

    Cost per blog post and per month: freelancer, agency, in-house, AI tools or Balzac. Your numbers, editable.

    Let Balzac write the articles.

    Type your website. Balzac finds the searches you can win, writes articles built to rank on Google and get cited by ChatGPT, and publishes them for you.

    3 free articles · No card needed