99 Popular Sites' robots.txt vs 16 AI Crawlers (Oct 2026)
We took 99 popular websites and checked each one’s public robots.txt for 16 AI crawlers. This post reports what we found, how we measured it, and the full per-site data. It is a snapshot taken on 2026-10-06, and it describes what the files say, nothing more.
Headline findings
Percentages below are of the 93 sites whose robots.txt we could read, not of all 99. Of the 99 sites, 93 had a readable robots.txt, 1 (khanacademy.org) had no robots.txt, and 5 could not be read reliably (godaddy.com, linkedin.com, mayoclinic.org, stackoverflow.com, udemy.com). We left those six out of every percentage.
- 55 of 93 sites (59%) block none of the 16 crawlers. The other 38 block at least one.
- 37 of 93 (40%) block at least one training crawler (GPTBot, ClaudeBot, CCBot, Google-Extended, Applebot-Extended, Meta-ExternalAgent or Bytespider).
- 23 of 93 (25%) block at least one AI search crawler (OAI-SearchBot, Claude-SearchBot or PerplexityBot). Training crawlers are blocked more often than search crawlers.
- 14 sites block GPTBot but allow OAI-SearchBot: airbnb.com, bloomberg.com, canva.com, ebay.com, figma.com, forbes.com, glassdoor.com, healthline.com, medium.com, reuters.com, techcrunch.com, tripadvisor.com, webmd.com, zalando.com. Only 1 site did the reverse (facebook.com blocks OAI-SearchBot but not GPTBot).
- 11 sites block all three AI search crawlers: amazon.com, cnn.com, imdb.com, instagram.com, nytimes.com, pinterest.com, quora.com, reddit.com, tiktok.com, x.com, yelp.com.
- 1 site blocks all 16 crawlers, including Googlebot and Bingbot: reddit.com.
- llms.txt: 29 of 99 sites publish one. 24 of those 29 are in our software and AI companies group. See the llms.txt section below.
By category
We assigned each site to one of five groups ourselves. The assignment is approximate and some sites could fit in more than one group, so read the rows as rough patterns rather than precise industry statistics. Percentages are of readable robots.txt files within each group; the llms.txt column counts all listed sites in the group.
| Category | Readable | Blocks GPTBot | Blocks OAI-SearchBot | Blocks ClaudeBot | Blocks any training bot | Has llms.txt |
|---|---|---|---|---|---|---|
| News and media | 19 | 11 (58%) | 4 (21%) | 17 (89%) | 18 (95%) | 0 of 20 |
| Social and community | 9 | 6 (67%) | 7 (78%) | 6 (67%) | 7 (78%) | 0 of 11 |
| Shopping and travel | 15 | 6 (40%) | 2 (13%) | 6 (40%) | 6 (40%) | 4 of 15 |
| Software and AI companies | 44 | 2 (5%) | 0 (0%) | 2 (5%) | 3 (7%) | 24 of 45 |
| Other | 6 | 1 (17%) | 0 (0%) | 1 (17%) | 3 (50%) | 1 of 8 |
Group sizes are small (6 to 44 readable sites), so a single site moves a percentage by several points.
By crawler
How many of the 93 readable sites block each of the 16 crawlers. A site counts as blocking a crawler if its robots.txt disallows path / for that user-agent (see the method below). “Training”, “Search” and “User-triggered” are our labels based on how each operator describes the crawler; Amazonbot is listed as “Other”.
| Crawler | Operator | Role | Sites blocking |
|---|---|---|---|
| Bytespider | ByteDance | Training | 33 (35%) |
| ClaudeBot | Anthropic | Training | 32 (34%) |
| CCBot | Common Crawl | Training | 31 (33%) |
| Applebot-Extended | Apple | Training | 30 (32%) |
| Meta-ExternalAgent | Meta | Training | 28 (30%) |
| GPTBot | OpenAI | Training | 26 (28%) |
| Amazonbot | Amazon | Other | 23 (25%) |
| Google-Extended | Training | 23 (25%) | |
| PerplexityBot | Perplexity | Search | 22 (24%) |
| Claude-SearchBot | Anthropic | Search | 18 (19%) |
| Claude-User | Anthropic | User-triggered | 18 (19%) |
| Perplexity-User | Perplexity | User-triggered | 18 (19%) |
| ChatGPT-User | OpenAI | User-triggered | 15 (16%) |
| OAI-SearchBot | OpenAI | Search | 13 (14%) |
| Bingbot | Microsoft | Classic search | 1 (1%) |
| Googlebot | Classic search | 1 (1%) |
Googlebot and Bingbot were included as a reference point for classic search. 1 site blocked Googlebot and 1 blocked Bingbot.
llms.txt
29 of 99 sites (29%) serve an llms.txt file. Most are software companies: 24 of the 45 in our software and AI companies group, against 4 in shopping and travel, 1 in other, and none in news and media or social and community.
A caution on interpreting this. llms.txt is a proposal, not a standard, and no major AI search engine promises to read it. Publishing one is not the same as being cited, and it does not change what robots.txt allows. We report the count because it was cheap to check, not because it shows a site is better prepared.
Method
- The free AI crawler checker read each site’s public robots.txt on 2026-10-06, following the parsing rules in RFC 9309.
- We tested path
/only. A crawler is counted as blocked when the rules that apply to it disallow/. Rules that only cover other paths were not tested. - The checker reads robots.txt rules only. A block at a CDN or firewall level is invisible to it, so a site counted as “not blocking” could still refuse these crawlers in practice.
- Sites can serve different robots.txt files to different visitors, for example by country or by user-agent. We saw one version per site.
- This is a snapshot. Files change, and some of these may already differ.
- We chose the sites for popularity and spread across industries, not at random, so the results do not describe the web as a whole.
- Six sites are excluded from percentages: khanacademy.org returned no robots.txt, and godaddy.com, linkedin.com, mayoclinic.org, stackoverflow.com, udemy.com could not be read reliably.
What to do with this
A robots.txt that blocks training crawlers but allows search crawlers is a choice some sites make: 14 of the 93 readable sites here block GPTBot while allowing OAI-SearchBot. It is one option among several, and this study does not say which option is right for any site.
The practical step is to know what your own file says. Run your domain through the AI crawler checker to see which of the 16 crawlers your robots.txt blocks today. If you want to change it, the robots.txt generator can draft a file, and the robots.txt checker can test specific URLs against your rules. If you use a CDN or firewall, check its bot settings separately, since robots.txt does not show them.
Download the data
Download the data as CSV. Columns: domain, robots_txt (found, missing or unreachable), llms_txt (yes or no), blocked_bots (semicolon-separated).
Appendix: all 99 sites
“Blocked” shows the number of the 16 crawlers blocked, followed by their names. “n/a” means robots.txt could not be evaluated for that site.
| Domain | robots.txt | Blocked crawlers | llms.txt |
|---|---|---|---|
| adobe.com | Read | 0 | yes |
| ahrefs.com | Read | 0 | no |
| airbnb.com | Read | 3: GPTBot, ClaudeBot, Applebot-Extended | no |
| amazon.com | Read | 12: OAI-SearchBot, ChatGPT-User, GPTBot, Claude-SearchBot, Claude-User, ClaudeBot, PerplexityBot, Perplexity-User, Google-Extended, Meta-ExternalAgent, CCBot, Bytespider | no |
| anthropic.com | Read | 0 | no |
| asana.com | Read | 0 | yes |
| atlassian.com | Read | 0 | yes |
| bbc.com | Read | 12: OAI-SearchBot, ChatGPT-User, GPTBot, ClaudeBot, PerplexityBot, Perplexity-User, Google-Extended, Applebot-Extended, Meta-ExternalAgent, CCBot, Bytespider, Amazonbot | no |
| bestbuy.com | Read | 0 | no |
| bloomberg.com | Read | 9: GPTBot, ClaudeBot, PerplexityBot, Google-Extended, Applebot-Extended, Meta-ExternalAgent, CCBot, Bytespider, Amazonbot | no |
| booking.com | Read | 0 | no |
| canva.com | Read | 6: GPTBot, ClaudeBot, Applebot-Extended, Meta-ExternalAgent, CCBot, Bytespider | no |
| cloudflare.com | Read | 0 | yes |
| cnn.com | Read | 13: OAI-SearchBot, ChatGPT-User, GPTBot, Claude-SearchBot, Claude-User, ClaudeBot, PerplexityBot, Perplexity-User, Google-Extended, Applebot-Extended, CCBot, Bytespider, Amazonbot | no |
| copy.ai | Read | 0 | no |
| coursera.org | Read | 1: Meta-ExternalAgent | yes |
| craigslist.org | Read | 0 | no |
| digitalocean.com | Read | 0 | no |
| dropbox.com | Read | 0 | yes |
| ebay.com | Read | 8: GPTBot, ClaudeBot, PerplexityBot, Applebot-Extended, Meta-ExternalAgent, CCBot, Bytespider, Amazonbot | no |
| etsy.com | Read | 0 | yes |
| expedia.com | Read | 0 | yes |
| facebook.com | Read | 8: OAI-SearchBot, ChatGPT-User, Claude-SearchBot, Claude-User, Perplexity-User, Meta-ExternalAgent, CCBot, Bytespider | no |
| figma.com | Read | 5: GPTBot, ClaudeBot, Google-Extended, CCBot, Bytespider | no |
| forbes.com | Read | 8: GPTBot, ClaudeBot, PerplexityBot, Applebot-Extended, Meta-ExternalAgent, CCBot, Bytespider, Amazonbot | no |
| github.com | Read | 1: Bytespider | yes |
| gitlab.com | Read | 0 | no |
| glassdoor.com | Read | 7: GPTBot, ClaudeBot, Google-Extended, Applebot-Extended, CCBot, Bytespider, Amazonbot | no |
| godaddy.com | Could not be read | n/a | no |
| grammarly.com | Read | 0 | no |
| healthline.com | Read | 7: GPTBot, ClaudeBot, Applebot-Extended, Meta-ExternalAgent, CCBot, Bytespider, Amazonbot | no |
| heroku.com | Read | 0 | yes |
| hubspot.com | Read | 0 | yes |
| huggingface.co | Read | 0 | no |
| ikea.com | Read | 0 | no |
| imdb.com | Read | 13: OAI-SearchBot, ChatGPT-User, GPTBot, Claude-SearchBot, Claude-User, ClaudeBot, PerplexityBot, Perplexity-User, Google-Extended, Applebot-Extended, Meta-ExternalAgent, CCBot, Bytespider | no |
| indeed.com | Read | 0 | no |
| instagram.com | Read | 14: OAI-SearchBot, ChatGPT-User, GPTBot, Claude-SearchBot, Claude-User, ClaudeBot, PerplexityBot, Perplexity-User, Google-Extended, Applebot-Extended, Meta-ExternalAgent, CCBot, Bytespider, Amazonbot | no |
| intercom.com | Read | 0 | yes |
| investopedia.com | Read | 11: Claude-SearchBot, Claude-User, ClaudeBot, PerplexityBot, Perplexity-User, Google-Extended, Applebot-Extended, Meta-ExternalAgent, CCBot, Bytespider, Amazonbot | no |
| jasper.ai | Read | 0 | yes |
| khanacademy.org | No robots.txt | n/a | no |
| linkedin.com | Could not be read | n/a | no |
| mailchimp.com | Read | 0 | no |
| mayoclinic.org | Could not be read | n/a | no |
| medium.com | Read | 6: GPTBot, ClaudeBot, Applebot-Extended, Meta-ExternalAgent, Bytespider, Amazonbot | no |
| mistral.ai | Read | 0 | yes |
| monday.com | Read | 0 | yes |
| moz.com | Read | 0 | no |
| nerdwallet.com | Read | 1: Google-Extended | no |
| netflix.com | Read | 6: Perplexity-User, Applebot-Extended, Meta-ExternalAgent, CCBot, Bytespider, Amazonbot | no |
| netlify.com | Read | 0 | yes |
| nike.com | Read | 0 | no |
| notion.so | Read | 1: Amazonbot | yes |
| nytimes.com | Read | 13: OAI-SearchBot, ChatGPT-User, GPTBot, Claude-SearchBot, Claude-User, ClaudeBot, PerplexityBot, Perplexity-User, Google-Extended, Applebot-Extended, Meta-ExternalAgent, CCBot, Bytespider | no |
| openai.com | Read | 0 | no |
| paypal.com | Read | 0 | yes |
| perplexity.ai | Read | 0 | no |
| pinterest.com | Read | 14: OAI-SearchBot, ChatGPT-User, GPTBot, Claude-SearchBot, Claude-User, ClaudeBot, PerplexityBot, Perplexity-User, Google-Extended, Applebot-Extended, Meta-ExternalAgent, CCBot, Bytespider, Amazonbot | no |
| quora.com | Read | 9: OAI-SearchBot, ChatGPT-User, GPTBot, Claude-SearchBot, Claude-User, ClaudeBot, PerplexityBot, Applebot-Extended, Bytespider | no |
| reddit.com | Read | 16: OAI-SearchBot, ChatGPT-User, GPTBot, Claude-SearchBot, Claude-User, ClaudeBot, PerplexityBot, Perplexity-User, Googlebot, Google-Extended, Bingbot, Applebot-Extended, Meta-ExternalAgent, CCBot, Bytespider, Amazonbot | no |
| reuters.com | Read | 12: GPTBot, Claude-SearchBot, Claude-User, ClaudeBot, PerplexityBot, Perplexity-User, Google-Extended, Applebot-Extended, Meta-ExternalAgent, CCBot, Bytespider, Amazonbot | no |
| revolut.com | Read | 0 | no |
| salesforce.com | Read | 0 | no |
| semrush.com | Read | 0 | yes |
| sendgrid.com | Read | 0 | no |
| shopify.com | Read | 0 | yes |
| slack.com | Read | 0 | yes |
| spotify.com | Read | 0 | no |
| squarespace.com | Read | 0 | yes |
| stackoverflow.com | Could not be read | n/a | no |
| stripe.com | Read | 0 | yes |
| substack.com | Read | 0 | no |
| surferseo.com | Read | 0 | no |
| target.com | Read | 0 | yes |
| techcrunch.com | Read | 7: ChatGPT-User, GPTBot, ClaudeBot, Google-Extended, Applebot-Extended, CCBot, Bytespider | no |
| theguardian.com | Read | 9: Claude-SearchBot, Claude-User, ClaudeBot, PerplexityBot, Applebot-Extended, Meta-ExternalAgent, CCBot, Bytespider, Amazonbot | no |
| theverge.com | Read | 11: ChatGPT-User, Claude-SearchBot, Claude-User, ClaudeBot, PerplexityBot, Perplexity-User, Google-Extended, Applebot-Extended, Meta-ExternalAgent, CCBot, Bytespider | no |
| tiktok.com | Read | 13: OAI-SearchBot, ChatGPT-User, GPTBot, Claude-SearchBot, Claude-User, ClaudeBot, PerplexityBot, Perplexity-User, Google-Extended, Applebot-Extended, Meta-ExternalAgent, CCBot, Bytespider | no |
| tripadvisor.com | Read | 8: GPTBot, ClaudeBot, Google-Extended, Applebot-Extended, Meta-ExternalAgent, CCBot, Bytespider, Amazonbot | no |
| udemy.com | Could not be read | n/a | no |
| vercel.com | Read | 0 | yes |
| walmart.com | Read | 0 | no |
| washingtonpost.com | Read | 7: ClaudeBot, PerplexityBot, Applebot-Extended, Meta-ExternalAgent, CCBot, Bytespider, Amazonbot | no |
| webflow.com | Read | 0 | yes |
| webmd.com | Read | 3: GPTBot, ClaudeBot, CCBot | no |
| wikipedia.org | Read | 0 | no |
| wired.com | Read | 11: Claude-SearchBot, Claude-User, ClaudeBot, PerplexityBot, Perplexity-User, Google-Extended, Applebot-Extended, Meta-ExternalAgent, CCBot, Bytespider, Amazonbot | no |
| wise.com | Read | 0 | no |
| wix.com | Read | 0 | yes |
| wordpress.com | Read | 0 | yes |
| writesonic.com | Read | 0 | yes |
| wsj.com | Read | 11: Claude-SearchBot, Claude-User, ClaudeBot, PerplexityBot, Perplexity-User, Google-Extended, Applebot-Extended, Meta-ExternalAgent, CCBot, Bytespider, Amazonbot | no |
| x.com | Read | 14: OAI-SearchBot, ChatGPT-User, GPTBot, Claude-SearchBot, Claude-User, ClaudeBot, PerplexityBot, Perplexity-User, Google-Extended, Applebot-Extended, Meta-ExternalAgent, CCBot, Bytespider, Amazonbot | no |
| yelp.com | Read | 14: OAI-SearchBot, ChatGPT-User, GPTBot, Claude-SearchBot, Claude-User, ClaudeBot, PerplexityBot, Perplexity-User, Google-Extended, Applebot-Extended, Meta-ExternalAgent, CCBot, Bytespider, Amazonbot | no |
| youtube.com | Read | 0 | no |
| zalando.com | Read | 8: GPTBot, ClaudeBot, Google-Extended, Applebot-Extended, Meta-ExternalAgent, CCBot, Bytespider, Amazonbot | no |
| zendesk.com | Read | 0 | no |
| zoom.us | Read | 0 | no |