TidyToolsAI Crawler Index · data as of · updated daily

AI Crawler Index · category

Do news websites block AI crawlers?

As of 2026-10-01, 43.7% of the 126 news websites whose robots.txt we could read block GPTBot, 49.2% block ClaudeBot and 45.2% block PerplexityBot (all sites: 17%, 16.3%, 12.7%). 10 of 139 publish an llms.txt file.

Block rate by crawler: news vs all sites

CrawlerOperatorNews & mediaAll sites
CCBotCommon Crawl58.7%19.5%
BytespiderByteDance52.4%17.7%
ClaudeBotAnthropic49.2%16.3%
PerplexityBotPerplexity45.2%12.7%
GPTBotOpenAI43.7%17%
Applebot-ExtendedApple43.7%14.3%
AmazonbotAmazon42.9%13.1%
cohere-aiCohere42.1%13.7%
Google-ExtendedGoogle41.3%14%
Meta-ExternalAgentMeta38.9%14.4%
ChatGPT-UserOpenAI28.6%9.6%
Claude-SearchBotAnthropic28.6%8.1%
Perplexity-UserPerplexity27.8%8.9%
Claude-UserAnthropic27%8.3%
meta-externalfetcherMeta27%8.7%
DuckAssistBotDuckDuckGo23.8%8.1%
MistralAI-UserMistral23.8%8.4%
OAI-SearchBotOpenAI21.4%6.7%
Google-CloudVertexBotGoogle16.7%7.2%
meta-webindexerMeta11.9%6.3%
Amzn-SearchBotAmazon11.9%5.9%
Amzn-UserAmazon10.3%5.5%
ApplebotApple7.9%2.6%
Google-AgentGoogle5.6%4.5%
MistralAI-IndexMistral5.6%4.7%
OAI-AdsBotOpenAI2.4%4.3%
Google-GeminiNotebookGoogle2.4%4.3%
MistralAI-TrainingMistral2.4%4.7%
meta-externaladsMeta1.6%3.8%

All 139 news sites in the index

✕ blocked · ◐ partially blocked (some paths) · ✓ allowed · ? robots.txt unreadable. "#" is the position in our popularity-ordered sample.

#SitePolicyGPTBotOAI-SearchChatGPT-UserClaudeBotClaude-SearchPerplexityGoogle-Ext.CCBotllms.txt
65nytimes.comBlocks some AI searchblockedblockedblockedblockedblockedblockedblockedblocked–
83medium.comBlocks training onlyblockedallowedallowedblockedallowedallowedallowedallowed–
84cnn.comUnreadableunknownunknownunknownunknownunknownunknownunknownunknown–
85theguardian.comBlocks some AI searchallowedallowedallowedblockedblockedblockedallowedblocked–
87t-online.deBlocks some AI searchblockedallowedallowedblockedallowedblockedpartialblocked–
91forbes.comBlocks some AI searchblockedallowedallowedblockedallowedblockedallowedblocked–
92bbc.comBlocks some AI searchblockedblockedblockedblockedallowedblockedblockedblocked–
119reuters.comBlocks some AI searchblockedpartialpartialblockedblockedblockedblockedblocked–
132weather.comBlocks some AI searchallowedallowedallowedallowedallowedallowedallowedblockedyes
156tradingview.comOpenallowedpartialpartialpartialallowedpartialpartialallowed–
165bloomberg.comBlocks some AI searchblockedpartialpartialblockedpartialblockedblockedblocked–
167wsj.comBlocks some AI searchpartialpartialpartialblockedblockedblockedblockedblocked–
171espn.comBlocks some AI searchblockedallowedblockedallowedallowedallowedblockedblocked–
173globo.comBlocks training onlyblockedallowedallowedallowedallowedallowedblockedblocked–
192substack.comOpenallowedallowedallowedallowedallowedallowedallowedallowed–
195businessinsider.comBlocks training onlyallowedallowedallowedblockedallowedallowedallowedblocked–
207cnbc.comBlocks some AI searchblockedblockedblockedblockedallowedblockedblockedblocked–
208statista.comOpenpartialallowedallowedpartialallowedpartialpartialpartial–
222indiatimes.comBlocks training onlyallowedallowedallowedallowedallowedallowedallowedallowed–
224foxnews.comOpenallowedallowedallowedallowedallowedallowedallowedallowedyes
237ft.comBlocks some AI searchallowedallowedallowedblockedallowedblockedblockedblocked–
240wired.comBlocks some AI searchallowedallowedallowedblockedblockedblockedblockedblocked–
247time.comOpenallowedallowedallowedallowedallowedallowedallowedallowed–
248uol.com.brBlocks some AI searchallowedallowedallowedblockedblockedblockedblockedblocked–
257usatoday.comBlocks some AI searchblockedblockedblockedblockedblockedblockedblockedblocked–
271telegraph.co.ukBlocks some AI searchblockedblockedblockedblockedblockedblockedallowedblocked–
275accuweather.comUnreadableunknownunknownunknownunknownunknownunknownunknownunknown–
281techcrunch.comBlocks some AI searchblockedallowedblockedblockedallowedallowedblockedblocked–
292elpais.comBlocks some AI searchallowedallowedallowedblockedblockedblockedallowedblocked–
309dailymail.comBlocks some AI searchblockedblockedblockedblockedblockedblockedblockedblocked–
312cbsnews.comBlocks training onlyblockedallowedallowedallowedallowedallowedallowedallowed–
313lemonde.frBlocks some AI searchallowedallowedallowedblockedblockedallowedblockedblocked–
323cnet.comBlocks some AI searchblockedblockedblockedblockedblockedblockedallowedblocked–
324independent.co.ukOpenallowedallowedallowedallowedallowedallowedallowedallowed–
329wp.plBlocks training onlyblockedallowedallowedallowedallowedallowedallowedblocked–
335bild.deBlocks some AI searchallowedallowedallowedallowedallowedblockedallowedblocked–
336mlb.comOpenallowedallowedallowedallowedallowedallowedallowedallowed–
341nikkei.comUnreadableunknownunknownunknownunknownunknownunknownunknownunknown–
351apnews.comBlocks some AI searchblockedallowedallowedblockedblockedblockedallowedblocked–
354welt.deBlocks some AI searchallowedallowedallowedallowedallowedblockedallowedblocked–
356prnewswire.comOpenallowedallowedallowedallowedallowedallowedallowedallowedyes
360people.comBlocks some AI searchpartialpartialpartialblockedblockedblockedblockedblocked–
366lefigaro.frUnreadableunknownunknownunknownunknownunknownunknownunknownunknown–
376theverge.comBlocks some AI searchallowedallowedblockedblockedblockedblockedblockedblocked–
383focus.deUnreadableunknownunknownunknownunknownunknownunknownunknownunknown–
391hbr.orgBlocks training onlyblockedallowedallowedallowedallowedallowedblockedblocked–
397nbcnews.comBlocks some AI searchblockedblockedblockedblockedblockedblockedblockedblocked–
407latimes.comBlocks training onlyblockedallowedallowedblockedallowedallowedallowedallowed–
425onet.plBlocks some AI searchallowedallowedallowedallowedallowedallowedallowedallowed–
426repubblica.itBlocks some AI searchblockedallowedallowedallowedallowedblockedblockedblockedyes
429corriere.itBlocks some AI searchblockedallowedblockedblockedallowedallowedblockedblocked–
436investopedia.comBlocks some AI searchpartialpartialpartialblockedblockedblockedblockedblocked–
438elmundo.esBlocks some AI searchblockedblockedblockedallowedallowedblockedblockedblocked–
439nationalgeographic.comOpenallowedallowedallowedallowedallowedallowedallowedallowed–
441flashscore.comBlocks some AI searchallowedallowedallowedallowedallowedallowedallowedblocked–
456infobae.comOpenallowedallowedallowedallowedallowedallowedallowedallowed–
459theatlantic.comBlocks some AI searchallowedallowedallowedblockedblockedblockedblockedblocked–
470moneycontrol.comUnreadableunknownunknownunknownunknownunknownunknownunknownunknown–
472pbs.orgBlocks some AI searchblockedallowedallowedallowedallowedblockedallowedblocked–
473n-tv.deBlocks training onlyblockedallowedallowedallowedallowedallowedblockedblocked–
476ouest-france.frUnreadableunknownunknownunknownunknownunknownunknownunknownunknown–
482interia.plOpenallowedallowedallowedallowedallowedallowedallowedallowed–
483goal.comOpenallowedallowedallowedallowedallowedallowedallowedallowed–
485terra.com.brOpenallowedallowedallowedallowedallowedallowedallowedallowed–
486as.comBlocks some AI searchallowedallowedallowedblockedblockedblockedallowedblocked–
489bfmtv.comBlocks some AI searchblockedblockedblockedblockedblockedblockedblockedblocked–
490huffpost.comBlocks some AI searchblockedblockedblockedblockedblockedblockedblockedblocked–
494marca.comBlocks some AI searchblockedblockedblockedallowedallowedblockedblockedblocked–
502skysports.comBlocks training onlyblockedallowedallowedallowedallowedallowedallowedblocked–
506wetteronline.deOpenallowedallowedallowedallowedallowedallowedallowedallowed–
509aftonbladet.seBlocks some AI searchallowedpartialallowedblockedblockedblockedblockedblocked–
511gsmarena.comBlocks some AI searchblockedpartialblockedblockedblockedblockedallowedblocked–
512lanacion.com.arOpenallowedallowedallowedallowedallowedallowedallowedallowed–
513hltv.orgOpenallowedallowedallowedallowedallowedallowedallowedallowed–
518nypost.comBlocks some AI searchallowedallowedallowedblockedblockedblockedblockedblocked–
522clarin.comOpenallowedallowedallowedallowedallowedallowedallowedallowed–
524economist.comBlocks some AI searchblockedblockedblockedblockedblockedblockedblockedblocked–
525theconversation.comBlocks some AI searchblockedblockedblockedblockedblockedblockedblockedblocked–
530ad.nlBlocks some AI searchblockedblockedblockedblockedallowedblockedblockedblocked–
534wetter.comOpenallowedallowedallowedallowedallowedallowedallowedallowed–
536itmedia.co.jpNo robots.txtallowedallowedallowedallowedallowedallowedallowedallowed–
537nu.nlBlocks some AI searchblockedblockedblockedblockedallowedblockedblockedblocked–
538elconfidencial.comBlocks some AI searchblockedblockedblockedblockedallowedblockedblockedblocked–
541vg.noBlocks some AI searchallowedpartialallowedblockedblockedblockedblockedblocked–
544spiegel.deBlocks some AI searchblockedblockedblockedblockedblockedallowedallowedblocked–
551cricinfo.comUnreadableunknownunknownunknownunknownunknownunknownunknownunknown–
552lequipe.frUnreadableunknownunknownunknownunknownunknownunknownunknownunknown–
564rbc.ruOpenallowedallowedallowedallowedallowedallowedallowedallowed–
567gazzetta.itBlocks some AI searchblockedallowedallowedblockedallowedallowedblockedblocked–
576hln.beBlocks some AI searchblockedblockedblockedblockedallowedblockedblockedblocked–
582mundodeportivo.comBlocks training onlyallowedallowedallowedblockedallowedallowedallowedallowed–
586abc.net.auBlocks some AI searchblockedallowedblockedblockedallowedblockedblockedblocked–
587livescore.comBlocks training onlyallowedallowedpartialallowedallowedallowedallowedallowed–
594fortune.comBlocks training onlyallowedallowedallowedallowedallowedallowedblockedblocked–
597ladepeche.frBlocks some AI searchblockedallowedallowedblockedallowedblockedblockedblocked–
605cricbuzz.comUnreadableunknownunknownunknownunknownunknownunknownunknownunknown–
607actu.frBlocks some AI searchblockedblockedblockedblockedblockedblockedblockedblocked–
610ilfattoquotidiano.itOpenallowedallowedallowedallowedallowedallowedallowedallowed–
621meteofrance.comOpenallowedallowedallowedallowedallowedallowedallowedallowed–
633tenki.jpOpenallowedallowedallowedallowedallowedallowedallowedallowed–
634weathernews.jpOpenpartialallowedallowedpartialallowedpartialpartialallowed–
635varzesh3.comOpenallowedallowedallowedallowedallowedallowedallowedallowed–
636sport.esUnreadableunknownunknownunknownunknownunknownunknownunknownunknown–
642oricon.co.jpOpenallowedallowedallowedallowedallowedallowedallowedallowed–
644haberler.comOpenallowedallowedallowedallowedallowedallowedpartialallowedyes
648nikkansports.comBlocks some AI searchblockedallowedblockedallowedallowedallowedblockedblocked–
649businesswire.comUnreadableunknownunknownunknownunknownunknownunknownunknownunknown–
650huffingtonpost.comBlocks some AI searchblockedblockedblockedblockedblockedblockedblockedblocked–
653novinky.czOpenallowedallowedallowedallowedallowedallowedallowedallowed–
656dagbladet.noBlocks some AI searchblockedpartialpartialblockedpartialblockedblockedblocked–
658okdiario.comOpenallowedallowedallowedallowedallowedallowedallowedallowed–
661ilmeteo.itBlocks training onlyblockedpartialpartialblockedpartialpartialpartialblockedyes
666sapo.ptBlocks some AI searchblockedblockedblockedblockedblockedblockedblockedblocked–
671cronista.comOpenallowedallowedallowedallowedallowedallowedallowedallowed–
673dw.comBlocks some AI searchblockedblockedblockedblockedallowedblockedblockedblocked–
675iltalehti.fiBlocks some AI searchblockedblockedblockedblockedblockedblockedblockedblocked–
678protothema.grOpenallowedallowedallowedallowedallowedallowedallowedallowedyes
680aljazeera.comBlocks some AI searchblockedallowedblockedblockedallowedblockedallowedallowed–
681blic.rsOpenallowedallowedallowedallowedallowedallowedallowedallowed–
683jutarnji.hrOpenallowedallowedallowedallowedallowedallowedallowedallowed–
68424h.com.vnOpenallowedallowedallowedallowedallowedallowedallowedallowed–
685index.hrOpenallowedallowedallowedallowedallowedallowedallowedallowed–
686sozcu.com.trOpenallowedallowedallowedallowedallowedallowedallowedallowedyes
6873bmeteo.comOpenallowedallowedallowedallowedallowedallowedallowedallowed–
695jagran.comOpenallowedallowedallowedallowedallowedallowedallowedallowed–
701ekstrabladet.dkBlocks some AI searchblockedblockedblockedblockedblockedblockedblockedblocked–
708eenadu.netUnreadableunknownunknownunknownunknownunknownunknownunknownunknown–
712kooora.comOpenallowedallowedallowedallowedallowedallowedallowedallowed–
718sondakika.comOpenallowedallowedallowedallowedallowedallowedpartialallowedyes
750meteociel.frBlocks training onlyallowedallowedallowedblockedpartialallowedallowedallowed–
789sportfilm800.comBlocks all AI searchblockedblockedblockedblockedblockedblockedblockedblocked–
845newsweek.comBlocks some AI searchblockedallowedblockedblockedallowedblockedallowedblocked–
868arstechnica.comBlocks some AI searchallowedallowedallowedblockedblockedblockedblockedblocked–
870sciencedaily.comOpenallowedallowedallowedallowedallowedallowedallowedallowed–
872zdnet.comBlocks some AI searchblockedblockedblockedblockedblockedblockedallowedblocked–
931euronews.comBlocks training onlyblockedpartialpartialallowedallowedpartialblockedblocked–
1001washingtonpost.comBlocks some AI searchallowedallowedallowedblockedallowedblockedallowedblocked–
1002engadget.comOpenallowedallowedallowedallowedallowedallowedallowedallowed–
1003dev.toOpenallowedallowedallowedallowedallowedallowedallowedallowedyes

FAQ

Do news websites block AI crawlers?
As of 2026-10-01, 43.7% of the 126 news websites whose robots.txt we could read block GPTBot, 49.2% block ClaudeBot and 45.2% block PerplexityBot (all sites: 17%, 16.3%, 12.7%). 10 of 139 publish an llms.txt file.
Which news websites block GPTBot?
nytimes.com, medium.com, t-online.de, forbes.com, bbc.com, reuters.com, bloomberg.com, espn.com, globo.com, cnbc.com, and more (full list on this page).

Other categories