TidyToolsAI Crawler Index · data as of · updated daily

AI Crawler Index · category

Do news websites block AI crawlers?

As of 2026-11-16, 33.7% of the 392 news websites whose robots.txt we could read block GPTBot, 35.7% block ClaudeBot and 28.1% block PerplexityBot (all sites: 15.4%, 14.7%, 8.2%). 42 of 431 publish an llms.txt file.

Block rate by crawler: news vs all sites

CrawlerOperatorNews & mediaAll sites
CCBotCommon Crawl37.8%16.4%
BytespiderByteDance37.8%16.2%
ClaudeBotAnthropic35.7%14.7%
GPTBotOpenAI33.7%15.4%
Applebot-ExtendedApple32.7%13.4%
Meta-ExternalAgentMeta30.6%13.6%
AmazonbotAmazon29.3%13.2%
cohere-aiCohere29.3%12.7%
Google-ExtendedGoogle28.3%13.2%
PerplexityBotPerplexity28.1%8.2%
ChatGPT-UserOpenAI20.9%7.4%
meta-externalfetcherMeta20.2%6.5%
Perplexity-UserPerplexity19.9%6.1%
Claude-SearchBotAnthropic19.6%5.7%
Claude-UserAnthropic18.9%5.9%
DuckAssistBotDuckDuckGo18.6%5.9%
OAI-SearchBotOpenAI17.9%5.8%
MistralAI-UserMistral16.3%5.7%
Google-CloudVertexBotGoogle12.8%8.8%
meta-webindexerMeta12.5%5.1%
Amzn-SearchBotAmazon9.9%4.4%
Amzn-UserAmazon8.2%4.4%
ApplebotApple6.9%3.2%
MistralAI-TrainingMistral6.4%7.1%
MistralAI-IndexMistral6.1%3.6%
Google-AgentGoogle5.6%3.6%
OAI-AdsBotOpenAI4.1%3%
Google-GeminiNotebookGoogle4.1%3.3%
meta-externaladsMeta3.3%2.9%

Top 150 of 431 news sites in the index

✕ blocked · ◐ partially blocked (some paths) · ✓ allowed · ? robots.txt unreadable. "#" is the position in our popularity-ordered sample.

#SitePolicyGPTBotOAI-SearchChatGPT-UserClaudeBotClaude-SearchPerplexityGoogle-Ext.CCBotllms.txt
67nytimes.comBlocks some AI searchblockedblockedblockedblockedblockedblockedblockedblocked–
83medium.comBlocks training onlyblockedallowedallowedblockedallowedallowedallowedallowed–
84cnn.comUnreadableunknownunknownunknownunknownunknownunknownunknownunknown–
85theguardian.comBlocks some AI searchallowedallowedallowedblockedblockedblockedallowedblocked–
90t-online.deBlocks some AI searchblockedallowedallowedblockedallowedblockedpartialblocked–
94forbes.comBlocks some AI searchblockedallowedallowedblockedallowedblockedallowedblocked–
95bbc.comBlocks some AI searchblockedblockedblockedblockedblockedblockedblockedblocked–
121reuters.comBlocks some AI searchblockedpartialpartialblockedblockedblockedblockedblocked–
135weather.comBlocks some AI searchallowedallowedallowedallowedallowedallowedallowedblockedyes
159washingtonpost.comBlocks some AI searchallowedallowedallowedblockedallowedblockedallowedblocked–
160tradingview.comOpenallowedpartialpartialpartialallowedpartialpartialallowed–
168bloomberg.comBlocks some AI searchblockedpartialpartialblockedpartialblockedblockedblocked–
170wsj.comBlocks some AI searchpartialpartialpartialblockedblockedblockedblockedblocked–
174espn.comBlocks some AI searchblockedallowedblockedallowedallowedallowedblockedblocked–
177globo.comBlocks training onlyblockedallowedallowedallowedallowedallowedblockedblocked–
196substack.comOpenallowedallowedallowedallowedallowedallowedallowedallowedyes
199businessinsider.comBlocks training onlyallowedallowedallowedblockedallowedallowedallowedblockedyes
209cnbc.comBlocks some AI searchblockedblockedblockedblockedblockedblockedblockedblocked–
210statista.comOpenpartialallowedallowedpartialallowedpartialpartialpartial–
226indiatimes.comBlocks training onlyallowedallowedallowedallowedallowedallowedallowedallowed–
228foxnews.comOpenallowedallowedallowedallowedallowedallowedallowedallowedyes
242ft.comBlocks some AI searchallowedallowedallowedblockedallowedblockedblockedblocked–
244wired.comBlocks some AI searchallowedallowedallowedblockedblockedblockedblockedblocked–
250uol.com.brBlocks some AI searchallowedallowedallowedblockedblockedblockedblockedblocked–
251time.comOpenallowedallowedallowedallowedallowedallowedallowedallowed–
262usatoday.comBlocks some AI searchblockedblockedblockedblockedblockedblockedblockedblocked–
276telegraph.co.ukBlocks some AI searchblockedblockedblockedblockedblockedblockedpartialblocked–
279accuweather.comUnreadableunknownunknownunknownunknownunknownunknownunknownunknown–
284techcrunch.comBlocks training onlyblockedallowedallowedblockedallowedallowedallowedblocked–
296elpais.comBlocks some AI searchallowedallowedallowedblockedblockedblockedallowedblocked–
313dailymail.comBlocks some AI searchblockedblockedblockedblockedblockedblockedblockedblockedyes
316cbsnews.comBlocks training onlyblockedallowedallowedallowedallowedallowedallowedallowed–
317lemonde.frBlocks some AI searchallowedallowedallowedblockedblockedallowedblockedblocked–
327independent.co.ukOpenallowedallowedallowedallowedallowedallowedallowedallowed–
328cnet.comBlocks some AI searchblockedblockedblockedblockedblockedblockedallowedblocked–
336wp.plBlocks training onlyblockedallowedallowedallowedallowedallowedallowedblocked–
340mlb.comOpenallowedallowedallowedallowedallowedallowedallowedallowed–
342bild.deBlocks some AI searchallowedallowedallowedallowedallowedblockedallowedblocked–
346nikkei.comUnreadableunknownunknownunknownunknownunknownunknownunknownunknown–
355apnews.comBlocks some AI searchblockedallowedallowedblockedblockedblockedallowedblocked–
357welt.deBlocks some AI searchallowedallowedallowedallowedallowedblockedallowedblocked–
359prnewswire.comOpenallowedallowedallowedallowedallowedallowedallowedallowedyes
363people.comBlocks some AI searchpartialpartialpartialblockedblockedblockedblockedblocked–
369lefigaro.frBlocks some AI searchallowedallowedallowedblockedblockedallowedblockedblocked–
380theverge.comBlocks some AI searchallowedallowedblockedblockedblockedblockedblockedblocked–
394focus.deUnreadableunknownunknownunknownunknownunknownunknownunknownunknown–
397hbr.orgBlocks some AI searchblockedallowedallowedblockedallowedallowedblockedblocked–
401nbcnews.comBlocks some AI searchblockedblockedblockedblockedblockedblockedblockedblocked–
410latimes.comBlocks training onlyblockedallowedallowedblockedallowedallowedallowedallowed–
427onet.plBlocks some AI searchallowedallowedallowedallowedallowedallowedallowedallowed–
430repubblica.itBlocks some AI searchblockedallowedallowedallowedallowedblockedblockedblockedyes
433corriere.itBlocks some AI searchblockedblockedblockedblockedblockedblockedblockedblocked–
439investopedia.comBlocks some AI searchpartialpartialpartialblockedblockedblockedblockedblocked–
442elmundo.esBlocks some AI searchblockedblockedblockedblockedblockedblockedblockedblocked–
444nationalgeographic.comOpenallowedallowedallowedallowedallowedallowedallowedallowed–
449flashscore.comBlocks some AI searchallowedallowedallowedallowedallowedallowedallowedallowed–
463infobae.comOpenallowedallowedallowedallowedallowedallowedallowedallowed–
464theatlantic.comBlocks some AI searchpartialpartialpartialblockedblockedblockedblockedblocked–
475moneycontrol.comUnreadableunknownunknownunknownunknownunknownunknownunknownunknown–
478n-tv.deBlocks training onlyblockedallowedallowedblockedallowedallowedblockedblocked–
479pbs.orgUnreadableunknownunknownunknownunknownunknownunknownunknownunknown–
484ouest-france.frBlocks some AI searchblockedblockedblockedblockedblockedblockedblockedblocked–
488goal.comOpenallowedallowedallowedallowedallowedallowedallowedallowed–
490interia.plOpenallowedallowedallowedallowedallowedallowedallowedallowed–
491terra.com.brOpenallowedallowedallowedallowedallowedallowedallowedallowedyes
493as.comBlocks some AI searchallowedallowedallowedblockedblockedblockedallowedblocked–
495huffpost.comBlocks some AI searchblockedblockedblockedblockedblockedblockedblockedblocked–
496bfmtv.comBlocks some AI searchblockedblockedblockedblockedblockedblockedblockedblocked–
500marca.comBlocks some AI searchblockedblockedblockedblockedblockedblockedblockedblocked–
509skysports.comBlocks training onlyblockedallowedallowedallowedallowedallowedallowedblocked–
515aftonbladet.seBlocks some AI searchallowedpartialallowedblockedblockedblockedblockedblocked–
516wetteronline.deNo robots.txtallowedallowedallowedallowedallowedallowedallowedallowed–
518gsmarena.comOpenallowedallowedallowedallowedallowedallowedallowedallowed–
522lanacion.com.arOpenallowedallowedallowedallowedallowedallowedallowedallowed–
524hltv.orgUnreadableunknownunknownunknownunknownunknownunknownunknownunknown–
528nypost.comBlocks some AI searchallowedallowedallowedblockedblockedblockedblockedblocked–
531clarin.comOpenallowedallowedallowedallowedallowedallowedallowedallowed–
534economist.comBlocks some AI searchblockedblockedblockedblockedblockedblockedblockedblocked–
537theconversation.comBlocks some AI searchblockedblockedblockedblockedblockedblockedblockedblocked–
540ad.nlBlocks some AI searchblockedblockedblockedblockedallowedblockedblockedblocked–
543wetter.comOpenallowedallowedallowedallowedallowedallowedallowedallowed–
545itmedia.co.jpNo robots.txtallowedallowedallowedallowedallowedallowedallowedallowed–
547nu.nlBlocks some AI searchblockedblockedblockedblockedblockedblockedblockedblocked–
548elconfidencial.comBlocks some AI searchblockedblockedblockedblockedallowedblockedblockedblocked–
554vg.noBlocks some AI searchallowedpartialallowedblockedblockedblockedblockedblocked–
559cricinfo.comUnreadableunknownunknownunknownunknownunknownunknownunknownunknown–
561lequipe.frUnreadableunknownunknownunknownunknownunknownunknownunknownunknown–
574gazzetta.itBlocks some AI searchblockedblockedblockedblockedblockedblockedblockedblocked–
578rbc.ruOpenallowedallowedallowedallowedallowedallowedallowedallowed–
583hln.beBlocks some AI searchblockedblockedblockedblockedallowedblockedblockedblocked–
592mundodeportivo.comBlocks training onlyblockedallowedallowedblockedallowedallowedallowedblocked–
595abc.net.auBlocks some AI searchblockedallowedblockedblockedallowedblockedblockedblocked–
596livescore.comBlocks training onlyallowedallowedpartialallowedallowedallowedallowedallowed–
605ladepeche.frBlocks some AI searchblockedallowedallowedblockedallowedblockedblockedblocked–
612fortune.comBlocks training onlyallowedallowedallowedallowedallowedallowedallowedblocked–
613spiegel.deBlocks some AI searchblockedblockedblockedblockedblockedallowedallowedblocked–
616cricbuzz.comUnreadableunknownunknownunknownunknownunknownunknownunknownunknown–
618actu.frBlocks some AI searchblockedblockedblockedblockedblockedblockedblockedblocked–
621ilfattoquotidiano.itOpenallowedallowedallowedallowedallowedallowedallowedallowed–
630meteofrance.comOpenallowedallowedallowedallowedallowedallowedallowedallowed–
642tenki.jpOpenallowedallowedallowedallowedallowedallowedallowedallowed–
643weathernews.jpOpenpartialallowedallowedpartialallowedpartialpartialallowed–
644sapo.ptBlocks some AI searchblockedblockedblockedblockedblockedblockedblockedblocked–
646varzesh3.comUnreadableunknownunknownunknownunknownunknownunknownunknownunknown–
648sport.esUnreadableunknownunknownunknownunknownunknownunknownunknownunknown–
653oricon.co.jpOpenallowedallowedallowedallowedallowedallowedallowedallowed–
655haberler.comOpenallowedpartialallowedallowedpartialpartialpartialallowedyes
659nikkansports.comBlocks some AI searchblockedallowedblockedallowedallowedallowedblockedblocked–
660businesswire.comUnreadableunknownunknownunknownunknownunknownunknownunknownunknown–
662novinky.czOpenallowedallowedallowedallowedallowedallowedallowedallowed–
666huffingtonpost.comBlocks some AI searchblockedblockedblockedblockedblockedblockedblockedblocked–
668dagbladet.noBlocks some AI searchblockedblockedblockedblockedblockedblockedblockedblocked–
670okdiario.comOpenallowedallowedallowedallowedallowedallowedallowedallowed–
673ilmeteo.itBlocks training onlyblockedpartialpartialblockedpartialpartialpartialblockedyes
680dw.comBlocks some AI searchblockedblockedblockedblockedallowedblockedblockedblocked–
683cronista.comOpenallowedallowedallowedallowedallowedallowedallowedallowed–
684iltalehti.fiBlocks some AI searchblockedblockedblockedblockedblockedblockedblockedblocked–
687protothema.grOpenallowedallowedallowedallowedallowedallowedallowedallowedyes
692aljazeera.comBlocks some AI searchblockedallowedblockedblockedallowedblockedallowedallowed–
693blic.rsOpenallowedallowedallowedallowedallowedallowedallowedallowed–
694jutarnji.hrOpenallowedallowedallowedallowedallowedallowedallowedallowed–
69724h.com.vnOpenallowedallowedallowedallowedallowedallowedallowedallowed–
698index.hrOpenallowedallowedallowedallowedallowedallowedallowedallowed–
699sozcu.com.trOpenallowedallowedallowedallowedallowedallowedallowedallowedyes
7003bmeteo.comOpenallowedallowedallowedallowedallowedallowedallowedallowed–
708jagran.comOpenallowedallowedallowedallowedallowedallowedallowedallowed–
713ekstrabladet.dkBlocks some AI searchblockedblockedblockedblockedblockedblockedblockedblocked–
719eenadu.netUnreadableunknownunknownunknownunknownunknownunknownunknownunknown–
724kooora.comOpenallowedallowedallowedallowedallowedallowedallowedallowed–
726sondakika.comOpenallowedpartialpartialallowedpartialpartialpartialallowedyes
760meteociel.frBlocks training onlyallowedallowedallowedblockedpartialallowedallowedallowed–
802sportfilm800.comBlocks some AI searchblockedallowedallowedblockedallowedallowedblockedblocked–
864newsweek.comBlocks some AI searchblockedallowedblockedblockedallowedblockedallowedblocked–
886arstechnica.comBlocks some AI searchallowedallowedallowedblockedblockedblockedblockedblocked–
889zdnet.comBlocks some AI searchblockedblockedblockedblockedblockedblockedallowedblocked–
890sciencedaily.comOpenallowedallowedallowedallowedallowedallowedallowedallowed–
956euronews.comBlocks training onlyblockedpartialpartialallowedallowedpartialallowedblocked–
971sport24.grOpenallowedallowedallowedallowedallowedallowedallowedallowed–
1055globenewswire.comUnreadableunknownunknownunknownunknownunknownunknownunknownunknown–
1098posthog.comOpenallowedallowedallowedallowedallowedallowedallowedallowedyes
1104engadget.comOpenallowedallowedallowedallowedallowedallowedallowedallowed–
1113medicalnewstoday.comBlocks some AI searchblockedallowedallowedblockedallowedallowedallowedblocked–
1142generatepress.comOpenallowedallowedallowedallowedallowedallowedallowedallowed–
1287abcnews.comBlocks some AI searchblockedallowedblockedallowedallowedallowedblockedblocked–
1324express.co.ukBlocks some AI searchblockedblockedallowedblockedallowedblockedallowedblocked–
1327news.com.auBlocks some AI searchblockedblockedblockedblockedblockedblockedblockedblocked–
1345expressapisv2.netBlocks all AI searchblockedblockedblockedblockedblockedblockedblockedblocked–
1366chinadaily.com.cnNo robots.txtallowedallowedallowedallowedallowedallowedallowedallowed–
1425hindustantimes.comBlocks some AI searchallowedallowedallowedallowedallowedallowedpartialallowedyes
1428sportradar.comOpenallowedallowedallowedallowedallowedallowedallowedallowedyes

Sorted by popularity. Every site has its own page: all 10,000 sites (A-Z lists), or download the CSV.

FAQ

Do news websites block AI crawlers?
As of 2026-11-16, 33.7% of the 392 news websites whose robots.txt we could read block GPTBot, 35.7% block ClaudeBot and 28.1% block PerplexityBot (all sites: 15.4%, 14.7%, 8.2%). 42 of 431 publish an llms.txt file.
Which news websites block GPTBot?
nytimes.com, medium.com, t-online.de, forbes.com, bbc.com, reuters.com, bloomberg.com, espn.com, globo.com, cnbc.com, and more (full list on this page).

Other categories