AI Crawler Index · category
Do news websites block AI crawlers?
As of 2026-11-16, 33.7% of the 392 news websites whose robots.txt we could read block GPTBot, 35.7% block ClaudeBot and 28.1% block PerplexityBot (all sites: 15.4%, 14.7%, 8.2%). 42 of 431 publish an llms.txt file.
Block rate by crawler: news vs all sites
| Crawler | Operator | News & media | All sites |
|---|---|---|---|
| CCBot | Common Crawl | 37.8% | 16.4% |
| Bytespider | ByteDance | 37.8% | 16.2% |
| ClaudeBot | Anthropic | 35.7% | 14.7% |
| GPTBot | OpenAI | 33.7% | 15.4% |
| Applebot-Extended | Apple | 32.7% | 13.4% |
| Meta-ExternalAgent | Meta | 30.6% | 13.6% |
| Amazonbot | Amazon | 29.3% | 13.2% |
| cohere-ai | Cohere | 29.3% | 12.7% |
| Google-Extended | 28.3% | 13.2% | |
| PerplexityBot | Perplexity | 28.1% | 8.2% |
| ChatGPT-User | OpenAI | 20.9% | 7.4% |
| meta-externalfetcher | Meta | 20.2% | 6.5% |
| Perplexity-User | Perplexity | 19.9% | 6.1% |
| Claude-SearchBot | Anthropic | 19.6% | 5.7% |
| Claude-User | Anthropic | 18.9% | 5.9% |
| DuckAssistBot | DuckDuckGo | 18.6% | 5.9% |
| OAI-SearchBot | OpenAI | 17.9% | 5.8% |
| MistralAI-User | Mistral | 16.3% | 5.7% |
| Google-CloudVertexBot | 12.8% | 8.8% | |
| meta-webindexer | Meta | 12.5% | 5.1% |
| Amzn-SearchBot | Amazon | 9.9% | 4.4% |
| Amzn-User | Amazon | 8.2% | 4.4% |
| Applebot | Apple | 6.9% | 3.2% |
| MistralAI-Training | Mistral | 6.4% | 7.1% |
| MistralAI-Index | Mistral | 6.1% | 3.6% |
| Google-Agent | 5.6% | 3.6% | |
| OAI-AdsBot | OpenAI | 4.1% | 3% |
| Google-GeminiNotebook | 4.1% | 3.3% | |
| meta-externalads | Meta | 3.3% | 2.9% |
Top 150 of 431 news sites in the index
✕ blocked · ◐ partially blocked (some paths) · ✓ allowed · ? robots.txt unreadable. "#" is the position in our popularity-ordered sample.
| # | Site | Policy | GPTBot | OAI-Search | ChatGPT-User | ClaudeBot | Claude-Search | Perplexity | Google-Ext. | CCBot | llms.txt |
|---|---|---|---|---|---|---|---|---|---|---|---|
| 67 | nytimes.com | Blocks some AI search | blocked | blocked | blocked | blocked | blocked | blocked | blocked | blocked | – |
| 83 | medium.com | Blocks training only | blocked | allowed | allowed | blocked | allowed | allowed | allowed | allowed | – |
| 84 | cnn.com | Unreadable | unknown | unknown | unknown | unknown | unknown | unknown | unknown | unknown | – |
| 85 | theguardian.com | Blocks some AI search | allowed | allowed | allowed | blocked | blocked | blocked | allowed | blocked | – |
| 90 | t-online.de | Blocks some AI search | blocked | allowed | allowed | blocked | allowed | blocked | partial | blocked | – |
| 94 | forbes.com | Blocks some AI search | blocked | allowed | allowed | blocked | allowed | blocked | allowed | blocked | – |
| 95 | bbc.com | Blocks some AI search | blocked | blocked | blocked | blocked | blocked | blocked | blocked | blocked | – |
| 121 | reuters.com | Blocks some AI search | blocked | partial | partial | blocked | blocked | blocked | blocked | blocked | – |
| 135 | weather.com | Blocks some AI search | allowed | allowed | allowed | allowed | allowed | allowed | allowed | blocked | yes |
| 159 | washingtonpost.com | Blocks some AI search | allowed | allowed | allowed | blocked | allowed | blocked | allowed | blocked | – |
| 160 | tradingview.com | Open | allowed | partial | partial | partial | allowed | partial | partial | allowed | – |
| 168 | bloomberg.com | Blocks some AI search | blocked | partial | partial | blocked | partial | blocked | blocked | blocked | – |
| 170 | wsj.com | Blocks some AI search | partial | partial | partial | blocked | blocked | blocked | blocked | blocked | – |
| 174 | espn.com | Blocks some AI search | blocked | allowed | blocked | allowed | allowed | allowed | blocked | blocked | – |
| 177 | globo.com | Blocks training only | blocked | allowed | allowed | allowed | allowed | allowed | blocked | blocked | – |
| 196 | substack.com | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | yes |
| 199 | businessinsider.com | Blocks training only | allowed | allowed | allowed | blocked | allowed | allowed | allowed | blocked | yes |
| 209 | cnbc.com | Blocks some AI search | blocked | blocked | blocked | blocked | blocked | blocked | blocked | blocked | – |
| 210 | statista.com | Open | partial | allowed | allowed | partial | allowed | partial | partial | partial | – |
| 226 | indiatimes.com | Blocks training only | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 228 | foxnews.com | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | yes |
| 242 | ft.com | Blocks some AI search | allowed | allowed | allowed | blocked | allowed | blocked | blocked | blocked | – |
| 244 | wired.com | Blocks some AI search | allowed | allowed | allowed | blocked | blocked | blocked | blocked | blocked | – |
| 250 | uol.com.br | Blocks some AI search | allowed | allowed | allowed | blocked | blocked | blocked | blocked | blocked | – |
| 251 | time.com | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 262 | usatoday.com | Blocks some AI search | blocked | blocked | blocked | blocked | blocked | blocked | blocked | blocked | – |
| 276 | telegraph.co.uk | Blocks some AI search | blocked | blocked | blocked | blocked | blocked | blocked | partial | blocked | – |
| 279 | accuweather.com | Unreadable | unknown | unknown | unknown | unknown | unknown | unknown | unknown | unknown | – |
| 284 | techcrunch.com | Blocks training only | blocked | allowed | allowed | blocked | allowed | allowed | allowed | blocked | – |
| 296 | elpais.com | Blocks some AI search | allowed | allowed | allowed | blocked | blocked | blocked | allowed | blocked | – |
| 313 | dailymail.com | Blocks some AI search | blocked | blocked | blocked | blocked | blocked | blocked | blocked | blocked | yes |
| 316 | cbsnews.com | Blocks training only | blocked | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 317 | lemonde.fr | Blocks some AI search | allowed | allowed | allowed | blocked | blocked | allowed | blocked | blocked | – |
| 327 | independent.co.uk | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 328 | cnet.com | Blocks some AI search | blocked | blocked | blocked | blocked | blocked | blocked | allowed | blocked | – |
| 336 | wp.pl | Blocks training only | blocked | allowed | allowed | allowed | allowed | allowed | allowed | blocked | – |
| 340 | mlb.com | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 342 | bild.de | Blocks some AI search | allowed | allowed | allowed | allowed | allowed | blocked | allowed | blocked | – |
| 346 | nikkei.com | Unreadable | unknown | unknown | unknown | unknown | unknown | unknown | unknown | unknown | – |
| 355 | apnews.com | Blocks some AI search | blocked | allowed | allowed | blocked | blocked | blocked | allowed | blocked | – |
| 357 | welt.de | Blocks some AI search | allowed | allowed | allowed | allowed | allowed | blocked | allowed | blocked | – |
| 359 | prnewswire.com | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | yes |
| 363 | people.com | Blocks some AI search | partial | partial | partial | blocked | blocked | blocked | blocked | blocked | – |
| 369 | lefigaro.fr | Blocks some AI search | allowed | allowed | allowed | blocked | blocked | allowed | blocked | blocked | – |
| 380 | theverge.com | Blocks some AI search | allowed | allowed | blocked | blocked | blocked | blocked | blocked | blocked | – |
| 394 | focus.de | Unreadable | unknown | unknown | unknown | unknown | unknown | unknown | unknown | unknown | – |
| 397 | hbr.org | Blocks some AI search | blocked | allowed | allowed | blocked | allowed | allowed | blocked | blocked | – |
| 401 | nbcnews.com | Blocks some AI search | blocked | blocked | blocked | blocked | blocked | blocked | blocked | blocked | – |
| 410 | latimes.com | Blocks training only | blocked | allowed | allowed | blocked | allowed | allowed | allowed | allowed | – |
| 427 | onet.pl | Blocks some AI search | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 430 | repubblica.it | Blocks some AI search | blocked | allowed | allowed | allowed | allowed | blocked | blocked | blocked | yes |
| 433 | corriere.it | Blocks some AI search | blocked | blocked | blocked | blocked | blocked | blocked | blocked | blocked | – |
| 439 | investopedia.com | Blocks some AI search | partial | partial | partial | blocked | blocked | blocked | blocked | blocked | – |
| 442 | elmundo.es | Blocks some AI search | blocked | blocked | blocked | blocked | blocked | blocked | blocked | blocked | – |
| 444 | nationalgeographic.com | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 449 | flashscore.com | Blocks some AI search | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 463 | infobae.com | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 464 | theatlantic.com | Blocks some AI search | partial | partial | partial | blocked | blocked | blocked | blocked | blocked | – |
| 475 | moneycontrol.com | Unreadable | unknown | unknown | unknown | unknown | unknown | unknown | unknown | unknown | – |
| 478 | n-tv.de | Blocks training only | blocked | allowed | allowed | blocked | allowed | allowed | blocked | blocked | – |
| 479 | pbs.org | Unreadable | unknown | unknown | unknown | unknown | unknown | unknown | unknown | unknown | – |
| 484 | ouest-france.fr | Blocks some AI search | blocked | blocked | blocked | blocked | blocked | blocked | blocked | blocked | – |
| 488 | goal.com | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 490 | interia.pl | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 491 | terra.com.br | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | yes |
| 493 | as.com | Blocks some AI search | allowed | allowed | allowed | blocked | blocked | blocked | allowed | blocked | – |
| 495 | huffpost.com | Blocks some AI search | blocked | blocked | blocked | blocked | blocked | blocked | blocked | blocked | – |
| 496 | bfmtv.com | Blocks some AI search | blocked | blocked | blocked | blocked | blocked | blocked | blocked | blocked | – |
| 500 | marca.com | Blocks some AI search | blocked | blocked | blocked | blocked | blocked | blocked | blocked | blocked | – |
| 509 | skysports.com | Blocks training only | blocked | allowed | allowed | allowed | allowed | allowed | allowed | blocked | – |
| 515 | aftonbladet.se | Blocks some AI search | allowed | partial | allowed | blocked | blocked | blocked | blocked | blocked | – |
| 516 | wetteronline.de | No robots.txt | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 518 | gsmarena.com | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 522 | lanacion.com.ar | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 524 | hltv.org | Unreadable | unknown | unknown | unknown | unknown | unknown | unknown | unknown | unknown | – |
| 528 | nypost.com | Blocks some AI search | allowed | allowed | allowed | blocked | blocked | blocked | blocked | blocked | – |
| 531 | clarin.com | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 534 | economist.com | Blocks some AI search | blocked | blocked | blocked | blocked | blocked | blocked | blocked | blocked | – |
| 537 | theconversation.com | Blocks some AI search | blocked | blocked | blocked | blocked | blocked | blocked | blocked | blocked | – |
| 540 | ad.nl | Blocks some AI search | blocked | blocked | blocked | blocked | allowed | blocked | blocked | blocked | – |
| 543 | wetter.com | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 545 | itmedia.co.jp | No robots.txt | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 547 | nu.nl | Blocks some AI search | blocked | blocked | blocked | blocked | blocked | blocked | blocked | blocked | – |
| 548 | elconfidencial.com | Blocks some AI search | blocked | blocked | blocked | blocked | allowed | blocked | blocked | blocked | – |
| 554 | vg.no | Blocks some AI search | allowed | partial | allowed | blocked | blocked | blocked | blocked | blocked | – |
| 559 | cricinfo.com | Unreadable | unknown | unknown | unknown | unknown | unknown | unknown | unknown | unknown | – |
| 561 | lequipe.fr | Unreadable | unknown | unknown | unknown | unknown | unknown | unknown | unknown | unknown | – |
| 574 | gazzetta.it | Blocks some AI search | blocked | blocked | blocked | blocked | blocked | blocked | blocked | blocked | – |
| 578 | rbc.ru | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 583 | hln.be | Blocks some AI search | blocked | blocked | blocked | blocked | allowed | blocked | blocked | blocked | – |
| 592 | mundodeportivo.com | Blocks training only | blocked | allowed | allowed | blocked | allowed | allowed | allowed | blocked | – |
| 595 | abc.net.au | Blocks some AI search | blocked | allowed | blocked | blocked | allowed | blocked | blocked | blocked | – |
| 596 | livescore.com | Blocks training only | allowed | allowed | partial | allowed | allowed | allowed | allowed | allowed | – |
| 605 | ladepeche.fr | Blocks some AI search | blocked | allowed | allowed | blocked | allowed | blocked | blocked | blocked | – |
| 612 | fortune.com | Blocks training only | allowed | allowed | allowed | allowed | allowed | allowed | allowed | blocked | – |
| 613 | spiegel.de | Blocks some AI search | blocked | blocked | blocked | blocked | blocked | allowed | allowed | blocked | – |
| 616 | cricbuzz.com | Unreadable | unknown | unknown | unknown | unknown | unknown | unknown | unknown | unknown | – |
| 618 | actu.fr | Blocks some AI search | blocked | blocked | blocked | blocked | blocked | blocked | blocked | blocked | – |
| 621 | ilfattoquotidiano.it | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 630 | meteofrance.com | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 642 | tenki.jp | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 643 | weathernews.jp | Open | partial | allowed | allowed | partial | allowed | partial | partial | allowed | – |
| 644 | sapo.pt | Blocks some AI search | blocked | blocked | blocked | blocked | blocked | blocked | blocked | blocked | – |
| 646 | varzesh3.com | Unreadable | unknown | unknown | unknown | unknown | unknown | unknown | unknown | unknown | – |
| 648 | sport.es | Unreadable | unknown | unknown | unknown | unknown | unknown | unknown | unknown | unknown | – |
| 653 | oricon.co.jp | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 655 | haberler.com | Open | allowed | partial | allowed | allowed | partial | partial | partial | allowed | yes |
| 659 | nikkansports.com | Blocks some AI search | blocked | allowed | blocked | allowed | allowed | allowed | blocked | blocked | – |
| 660 | businesswire.com | Unreadable | unknown | unknown | unknown | unknown | unknown | unknown | unknown | unknown | – |
| 662 | novinky.cz | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 666 | huffingtonpost.com | Blocks some AI search | blocked | blocked | blocked | blocked | blocked | blocked | blocked | blocked | – |
| 668 | dagbladet.no | Blocks some AI search | blocked | blocked | blocked | blocked | blocked | blocked | blocked | blocked | – |
| 670 | okdiario.com | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 673 | ilmeteo.it | Blocks training only | blocked | partial | partial | blocked | partial | partial | partial | blocked | yes |
| 680 | dw.com | Blocks some AI search | blocked | blocked | blocked | blocked | allowed | blocked | blocked | blocked | – |
| 683 | cronista.com | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 684 | iltalehti.fi | Blocks some AI search | blocked | blocked | blocked | blocked | blocked | blocked | blocked | blocked | – |
| 687 | protothema.gr | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | yes |
| 692 | aljazeera.com | Blocks some AI search | blocked | allowed | blocked | blocked | allowed | blocked | allowed | allowed | – |
| 693 | blic.rs | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 694 | jutarnji.hr | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 697 | 24h.com.vn | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 698 | index.hr | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 699 | sozcu.com.tr | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | yes |
| 700 | 3bmeteo.com | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 708 | jagran.com | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 713 | ekstrabladet.dk | Blocks some AI search | blocked | blocked | blocked | blocked | blocked | blocked | blocked | blocked | – |
| 719 | eenadu.net | Unreadable | unknown | unknown | unknown | unknown | unknown | unknown | unknown | unknown | – |
| 724 | kooora.com | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 726 | sondakika.com | Open | allowed | partial | partial | allowed | partial | partial | partial | allowed | yes |
| 760 | meteociel.fr | Blocks training only | allowed | allowed | allowed | blocked | partial | allowed | allowed | allowed | – |
| 802 | sportfilm800.com | Blocks some AI search | blocked | allowed | allowed | blocked | allowed | allowed | blocked | blocked | – |
| 864 | newsweek.com | Blocks some AI search | blocked | allowed | blocked | blocked | allowed | blocked | allowed | blocked | – |
| 886 | arstechnica.com | Blocks some AI search | allowed | allowed | allowed | blocked | blocked | blocked | blocked | blocked | – |
| 889 | zdnet.com | Blocks some AI search | blocked | blocked | blocked | blocked | blocked | blocked | allowed | blocked | – |
| 890 | sciencedaily.com | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 956 | euronews.com | Blocks training only | blocked | partial | partial | allowed | allowed | partial | allowed | blocked | – |
| 971 | sport24.gr | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 1055 | globenewswire.com | Unreadable | unknown | unknown | unknown | unknown | unknown | unknown | unknown | unknown | – |
| 1098 | posthog.com | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | yes |
| 1104 | engadget.com | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 1113 | medicalnewstoday.com | Blocks some AI search | blocked | allowed | allowed | blocked | allowed | allowed | allowed | blocked | – |
| 1142 | generatepress.com | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 1287 | abcnews.com | Blocks some AI search | blocked | allowed | blocked | allowed | allowed | allowed | blocked | blocked | – |
| 1324 | express.co.uk | Blocks some AI search | blocked | blocked | allowed | blocked | allowed | blocked | allowed | blocked | – |
| 1327 | news.com.au | Blocks some AI search | blocked | blocked | blocked | blocked | blocked | blocked | blocked | blocked | – |
| 1345 | expressapisv2.net | Blocks all AI search | blocked | blocked | blocked | blocked | blocked | blocked | blocked | blocked | – |
| 1366 | chinadaily.com.cn | No robots.txt | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 1425 | hindustantimes.com | Blocks some AI search | allowed | allowed | allowed | allowed | allowed | allowed | partial | allowed | yes |
| 1428 | sportradar.com | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | yes |
Sorted by popularity. Every site has its own page: all 10,000 sites (A-Z lists), or download the CSV.
FAQ
- Do news websites block AI crawlers?
- As of 2026-11-16, 33.7% of the 392 news websites whose robots.txt we could read block GPTBot, 35.7% block ClaudeBot and 28.1% block PerplexityBot (all sites: 15.4%, 14.7%, 8.2%). 42 of 431 publish an llms.txt file.
- Which news websites block GPTBot?
- nytimes.com, medium.com, t-online.de, forbes.com, bbc.com, reuters.com, bloomberg.com, espn.com, globo.com, cnbc.com, and more (full list on this page).
Other categories
- AI GPTBot 10%
- E-commerce GPTBot 14.7%
- Education GPTBot 5.6%
- Entertainment GPTBot 19.6%
- Finance GPTBot 6.9%
- Government GPTBot 3%
- Other GPTBot 15.1%
- Reference GPTBot 27.9%
- Social GPTBot 37.3%
- Tech GPTBot 3.7%
- Telecom GPTBot 0%
- Travel GPTBot 15.4%