Cloudflare and AI crawlers
Cloudflare and AI crawlers,
what your site blocks
How to check what your Cloudflare setup blocks, as Cloudflare documents it. Verified from Cloudflare's docs on 2026-10-09.
Cloudflare can turn AI crawlers away before they ever reach your robots.txt. This page lists what Cloudflare documents about that: the AI Crawl Control and AI bot policy settings, the managed robots.txt, and how both differ from a plain robots.txt, with the wording taken from Cloudflare's pages.
What Cloudflare documents
| Topic | What Cloudflare says | Source |
|---|---|---|
| What AI Crawl Control is | “AI Crawl Control (formerly AI Audit) gives you visibility into which AI services are accessing your content.”Cloudflare marks it as available on all plans and lists four parts: manage AI crawlers, analyze AI traffic, track robots.txt and Pay Per Crawl. | Cloudflare docs Verified 2026-10-09 |
| Block or allow a crawler | “You can block an AI crawler to completely stop the AI crawler from scraping your webpage.”Cloudflare: you choose Block or Allow per crawler in the Actions column of the Crawlers table. When you block, you can set the response to 403 Forbidden or 402 Payment Required. | Cloudflare docs Verified 2026-10-09 |
| How a block is enforced | “When you block a crawler in AI Crawl Control, the system creates or updates a WAF custom rule on your zone to enforce that block.” | Cloudflare docs Verified 2026-10-09 |
| Behavior policies | “All Cloudflare customers can choose to block AI bots and agents based on their behavior.”Cloudflare groups bots as Search, Agent and Training. For each it offers Block (on all pages), Block on pages with ads, or Allow (do not block), under Security Settings > Configure AI bot policies. | Cloudflare docs Verified 2026-10-09 |
| The older Block AI bots setting | “This setting blocks verified bots that are classified as crawling for the purpose of AI training, as well as a number of unverified bots that behave similarly.”Cloudflare's page marks this setting as deprecating on September 15, 2026. | Cloudflare docs Verified 2026-10-09 |
| Defaults for new domains | “On September 15, 2026, Cloudflare will set updated defaults for new domains:”Cloudflare's page continues: bots classified as Training or as Agent will be blocked on pages that display ads, and Search will remain allowed. The page was last updated on Jul 1, 2026 and still used the future tense when read, so it does not tell us whether the change has already happened. | Cloudflare docs Verified 2026-10-09 |
| Managed robots.txt | “Bot Preference Sync generates managed robots.txt directives from your Search, Agent, and Training policies.”Cloudflare: if your origin already serves a robots.txt, it prepends its managed robots.txt before your existing one; if there is none, it creates one with managed Disallow rules for known AI crawlers. | Cloudflare docs Verified 2026-10-09 |
| A robots.txt only states a preference | “The file expresses your preferences, but it does not prevent crawlers from accessing your content at a technical level.”Cloudflare: robots.txt compliance is voluntary, and to enforce access restrictions you use AI Crawl Control to block selected AI crawlers. | Cloudflare docs Verified 2026-10-09 |
| robots.txt violations | “identifies AI crawlers that have requested paths explicitly disallowed by your robots.txt file”Cloudflare: the Track robots.txt view matches requests against your current rules, and violations are not logged in real time, so recently added rules can flag older requests. | Cloudflare docs Verified 2026-10-09 |
| Verified bots | “a bot or agent that Cloudflare has confirmed is transparent about who it is and what it does”Cloudflare: a verified bot must identify itself reliably and behave acceptably, including respecting robots.txt. Cloudflare's block options cover verified bots with the chosen behavior and also additional unverified bots in the same classifications. | Cloudflare docs Verified 2026-10-09 |
| Pay per crawl | “a feature of AI Crawl Control that enables site owners to control and monetize AI crawler access”Cloudflare says it is currently in closed beta. A crawler without payment intent receives an HTTP 402 Payment Required response with pricing. | Cloudflare docs Verified 2026-10-09 |
Why a robots.txt checker cannot see this
A robots.txt checker reads one public file and applies its rules. A Cloudflare block is a separate layer: Cloudflare says a block in AI Crawl Control creates or updates a WAF custom rule on your zone, and it says robots.txt on its own does not prevent crawlers from accessing your content at a technical level. So a site can show "allowed" in its robots.txt and still turn a crawler away at Cloudflare, or the reverse.
Our free AI crawler checker reads robots.txt only, and says that a CDN or firewall can still block crawlers it allows. To see what Cloudflare does for your site, look in your own Cloudflare dashboard: Security Settings > Configure AI bot policies, and AI Crawl Control.
Rules to remember
- Cloudflare says robots.txt compliance is voluntary, and that to enforce access restrictions you block crawlers in AI Crawl Control.
- Cloudflare's behavior policies are Search, Agent and Training, each with Block, Block on pages with ads, or Allow.
- Cloudflare's managed robots.txt is prepended to your own file when your origin serves one.
Questions
Can a robots.txt checker see that Cloudflare blocks AI bots?
No. A robots.txt checker reads only the file. Cloudflare's block runs at its network layer, through a WAF custom rule, so a crawler can be turned away even when robots.txt allows it. Our AI crawler checker says the same: it checks robots.txt only, and a CDN or firewall can still block crawlers it allows.
How is a Cloudflare block different from robots.txt?
Cloudflare says robots.txt expresses your preferences but does not prevent crawlers from accessing your content at a technical level, and that compliance is voluntary. Blocking in AI Crawl Control is the enforcement: Cloudflare creates or updates a WAF custom rule on your zone.
Does Cloudflare write to my robots.txt?
Cloudflare documents a managed robots.txt (Bot Preference Sync). If your origin serves a robots.txt it prepends its managed directives to it; if not, it creates one with Disallow rules for known AI crawlers. It is added when at least one policy is set to block or, for Training, Disallow.
What are the defaults for new domains?
Cloudflare's Block AI bots page says that on September 15, 2026 it will set updated defaults for new domains: Training and Agent bots blocked on pages that display ads, Search allowed. That page was last updated on Jul 1, 2026 and still used the future tense when we read it on 2026-10-09, so check your own dashboard.
Where do I check what my Cloudflare zone does?
Cloudflare points to Security Settings > Configure AI bot policies for behavior policies, and to AI Crawl Control for per-crawler Block and Allow actions. Look there, or at the WAF custom rules on the zone, because a public robots.txt tool cannot show them.
How current is this page?
Every row was read from Cloudflare's documentation on 2026-10-09. The pages showed last-updated dates between Apr 23, 2026 and Oct 8, 2026. Cloudflare can change its documentation, so treat the linked pages as the final word.
Sources
- AI Crawl Control (Cloudflare docs), read on 2026-10-09
- Manage AI crawlers (Cloudflare docs), read on 2026-10-09
- Track robots.txt (Cloudflare docs), read on 2026-10-09
- What is pay per crawl (Cloudflare docs), read on 2026-10-09
- Block AI bots and Configure AI bot policies (Cloudflare docs), read on 2026-10-09
- Managed robots.txt (Cloudflare docs), read on 2026-10-09
- Verified bots (Cloudflare docs), read on 2026-10-09
Let Balzac write the articles.
Type your website. Balzac finds the searches you can win, writes articles built to rank on Google and get cited by ChatGPT, and publishes them for you.
3 free articles · No card needed