Cloudflare and AI crawlers

Cloudflare and AI crawlers,
what your site blocks

How to check what your Cloudflare setup blocks, as Cloudflare documents it. Verified from Cloudflare's docs on 2026-10-09.

Cloudflare can turn AI crawlers away before they ever reach your robots.txt. This page lists what Cloudflare documents about that: the AI Crawl Control and AI bot policy settings, the managed robots.txt, and how both differ from a plain robots.txt, with the wording taken from Cloudflare's pages.

What Cloudflare documents

TopicWhat Cloudflare saysSource
What AI Crawl Control is “AI Crawl Control (formerly AI Audit) gives you visibility into which AI services are accessing your content.”Cloudflare marks it as available on all plans and lists four parts: manage AI crawlers, analyze AI traffic, track robots.txt and Pay Per Crawl. Cloudflare docs
Verified 2026-10-09
Block or allow a crawler “You can block an AI crawler to completely stop the AI crawler from scraping your webpage.”Cloudflare: you choose Block or Allow per crawler in the Actions column of the Crawlers table. When you block, you can set the response to 403 Forbidden or 402 Payment Required. Cloudflare docs
Verified 2026-10-09
How a block is enforced “When you block a crawler in AI Crawl Control, the system creates or updates a WAF custom rule on your zone to enforce that block.” Cloudflare docs
Verified 2026-10-09
Behavior policies “All Cloudflare customers can choose to block AI bots and agents based on their behavior.”Cloudflare groups bots as Search, Agent and Training. For each it offers Block (on all pages), Block on pages with ads, or Allow (do not block), under Security Settings > Configure AI bot policies. Cloudflare docs
Verified 2026-10-09
The older Block AI bots setting “This setting blocks verified bots that are classified as crawling for the purpose of AI training, as well as a number of unverified bots that behave similarly.”Cloudflare's page marks this setting as deprecating on September 15, 2026. Cloudflare docs
Verified 2026-10-09
Defaults for new domains “On September 15, 2026, Cloudflare will set updated defaults for new domains:”Cloudflare's page continues: bots classified as Training or as Agent will be blocked on pages that display ads, and Search will remain allowed. The page was last updated on Jul 1, 2026 and still used the future tense when read, so it does not tell us whether the change has already happened. Cloudflare docs
Verified 2026-10-09
Managed robots.txt “Bot Preference Sync generates managed robots.txt directives from your Search, Agent, and Training policies.”Cloudflare: if your origin already serves a robots.txt, it prepends its managed robots.txt before your existing one; if there is none, it creates one with managed Disallow rules for known AI crawlers. Cloudflare docs
Verified 2026-10-09
A robots.txt only states a preference “The file expresses your preferences, but it does not prevent crawlers from accessing your content at a technical level.”Cloudflare: robots.txt compliance is voluntary, and to enforce access restrictions you use AI Crawl Control to block selected AI crawlers. Cloudflare docs
Verified 2026-10-09
robots.txt violations “identifies AI crawlers that have requested paths explicitly disallowed by your robots.txt file”Cloudflare: the Track robots.txt view matches requests against your current rules, and violations are not logged in real time, so recently added rules can flag older requests. Cloudflare docs
Verified 2026-10-09
Verified bots “a bot or agent that Cloudflare has confirmed is transparent about who it is and what it does”Cloudflare: a verified bot must identify itself reliably and behave acceptably, including respecting robots.txt. Cloudflare's block options cover verified bots with the chosen behavior and also additional unverified bots in the same classifications. Cloudflare docs
Verified 2026-10-09
Pay per crawl “a feature of AI Crawl Control that enables site owners to control and monetize AI crawler access”Cloudflare says it is currently in closed beta. A crawler without payment intent receives an HTTP 402 Payment Required response with pricing. Cloudflare docs
Verified 2026-10-09

Read on 2026-10-09. Last-updated dates shown by Cloudflare: AI Crawl Control Aug 14, 2026; Manage AI crawlers Jul 28, 2026; Track robots.txt Apr 23, 2026; Pay per crawl Jul 28, 2026; Block AI bots Jul 1, 2026; Managed robots.txt Oct 8, 2026; Verified bots Oct 8, 2026. Notes under a quote are close paraphrases of the same page. This page covers only what Cloudflare documents; it does not say what any particular site has configured.

Why a robots.txt checker cannot see this

A robots.txt checker reads one public file and applies its rules. A Cloudflare block is a separate layer: Cloudflare says a block in AI Crawl Control creates or updates a WAF custom rule on your zone, and it says robots.txt on its own does not prevent crawlers from accessing your content at a technical level. So a site can show "allowed" in its robots.txt and still turn a crawler away at Cloudflare, or the reverse.

Our free AI crawler checker reads robots.txt only, and says that a CDN or firewall can still block crawlers it allows. To see what Cloudflare does for your site, look in your own Cloudflare dashboard: Security Settings > Configure AI bot policies, and AI Crawl Control.

Rules to remember

  • Cloudflare says robots.txt compliance is voluntary, and that to enforce access restrictions you block crawlers in AI Crawl Control.
  • Cloudflare's behavior policies are Search, Agent and Training, each with Block, Block on pages with ads, or Allow.
  • Cloudflare's managed robots.txt is prepended to your own file when your origin serves one.

Which crawler is which: AI crawler user agents. What the AI companies say about access to your site: what OpenAI, Anthropic, Perplexity and Microsoft say. Confirm what a robots.txt says with the robots.txt and sitemap checker.

Questions

Can a robots.txt checker see that Cloudflare blocks AI bots?

No. A robots.txt checker reads only the file. Cloudflare's block runs at its network layer, through a WAF custom rule, so a crawler can be turned away even when robots.txt allows it. Our AI crawler checker says the same: it checks robots.txt only, and a CDN or firewall can still block crawlers it allows.

How is a Cloudflare block different from robots.txt?

Cloudflare says robots.txt expresses your preferences but does not prevent crawlers from accessing your content at a technical level, and that compliance is voluntary. Blocking in AI Crawl Control is the enforcement: Cloudflare creates or updates a WAF custom rule on your zone.

Does Cloudflare write to my robots.txt?

Cloudflare documents a managed robots.txt (Bot Preference Sync). If your origin serves a robots.txt it prepends its managed directives to it; if not, it creates one with Disallow rules for known AI crawlers. It is added when at least one policy is set to block or, for Training, Disallow.

What are the defaults for new domains?

Cloudflare's Block AI bots page says that on September 15, 2026 it will set updated defaults for new domains: Training and Agent bots blocked on pages that display ads, Search allowed. That page was last updated on Jul 1, 2026 and still used the future tense when we read it on 2026-10-09, so check your own dashboard.

Where do I check what my Cloudflare zone does?

Cloudflare points to Security Settings > Configure AI bot policies for behavior policies, and to AI Crawl Control for per-crawler Block and Allow actions. Look there, or at the WAF custom rules on the zone, because a public robots.txt tool cannot show them.

How current is this page?

Every row was read from Cloudflare's documentation on 2026-10-09. The pages showed last-updated dates between Apr 23, 2026 and Oct 8, 2026. Cloudflare can change its documentation, so treat the linked pages as the final word.

Sources

Let Balzac write the articles.

Type your website. Balzac finds the searches you can win, writes articles built to rank on Google and get cited by ChatGPT, and publishes them for you.

3 free articles · No card needed