TidyToolsAI Crawler Index · data as of · updated daily

AI Crawler Index · category

Do entertainment websites block AI crawlers?

As of 2026-10-01, 13.2% of the 68 entertainment websites whose robots.txt we could read block GPTBot, 8.8% block ClaudeBot and 5.9% block PerplexityBot (all sites: 17%, 16.3%, 12.7%). 7 of 77 publish an llms.txt file.

Block rate by crawler: entertainment vs all sites

CrawlerOperatorEntertainmentAll sites
cohere-aiCohere16.2%13.7%
CCBotCommon Crawl14.7%19.5%
GPTBotOpenAI13.2%17%
Meta-ExternalAgentMeta13.2%14.4%
BytespiderByteDance13.2%17.7%
ClaudeBotAnthropic8.8%16.3%
Google-ExtendedGoogle8.8%14%
Applebot-ExtendedApple8.8%14.3%
Perplexity-UserPerplexity7.4%8.9%
Google-CloudVertexBotGoogle7.4%7.2%
meta-webindexerMeta7.4%6.3%
MistralAI-UserMistral7.4%8.4%
ChatGPT-UserOpenAI5.9%9.6%
PerplexityBotPerplexity5.9%12.7%
OAI-AdsBotOpenAI4.4%4.3%
Google-GeminiNotebookGoogle4.4%4.3%
meta-externalfetcherMeta4.4%8.7%
AmazonbotAmazon4.4%13.1%
Amzn-SearchBotAmazon4.4%5.9%
Amzn-UserAmazon4.4%5.5%
DuckAssistBotDuckDuckGo4.4%8.1%
MistralAI-IndexMistral4.4%4.7%
MistralAI-TrainingMistral4.4%4.7%
Claude-UserAnthropic2.9%8.3%
Google-AgentGoogle1.5%4.5%
meta-externaladsMeta1.5%3.8%
OAI-SearchBotOpenAI0%6.7%
Claude-SearchBotAnthropic0%8.1%
ApplebotApple0%2.6%

All 77 entertainment sites in the index

✕ blocked · ◐ partially blocked (some paths) · ✓ allowed · ? robots.txt unreadable. "#" is the position in our popularity-ordered sample.

#SitePolicyGPTBotOAI-SearchChatGPT-UserClaudeBotClaude-SearchPerplexityGoogle-Ext.CCBotllms.txt
5youtube.comOpenallowedallowedallowedallowedallowedallowedallowedallowed–
18netflix.comBlocks some AI searchpartialpartialpartialpartialpartialpartialpartialblocked–
28roblox.comOpenallowedallowedallowedallowedallowedallowedallowedallowed–
29spotify.comOpenallowedallowedallowedallowedallowedallowedallowedallowed–
36vimeo.comBlocks training onlypartialallowedallowedpartialallowedallowedpartialblocked–
50amazonvideo.comBlocks some AI searchallowedallowedblockedallowedallowedblockedallowedblocked–
51epicgames.comUnreadableunknownunknownunknownunknownunknownunknownunknownunknown–
80soundcloud.comOpenpartialpartialpartialpartialallowedpartialpartialpartial–
95twitch.tvOpenallowedallowedallowedallowedallowedallowedallowedallowed–
126steampowered.comOpenallowedallowedallowedallowedallowedallowedallowedallowed–
144bilibili.comOpenallowedallowedallowedallowedallowedallowedallowedallowedyes
148ea.comBlocks training onlyallowedallowedallowedallowedallowedallowedallowedallowed–
184dailymotion.comOpenallowedallowedallowedallowedallowedallowedallowedallowedyes
185steamcommunity.comOpenallowedallowedallowedallowedallowedallowedallowedallowed–
188supercell.comOpenallowedallowedallowedallowedallowedallowedallowedallowed–
214ivi.ruOpenallowedallowedallowedallowedallowedallowedallowedallowed–
262playstation.comOpenallowedallowedallowedallowedallowedallowedallowedallowedyes
273nintendo.comBlocks training onlyblockedallowedallowedblockedallowedallowedblockedblocked–
291primevideo.comBlocks some AI searchallowedallowedblockedallowedallowedblockedallowedblocked–
310kick.comOpenallowedallowedallowedallowedallowedallowedallowedallowed–
322itch.ioOpenallowedallowedallowedallowedallowedallowedallowedallowed–
326imgur.comOpenallowedallowedallowedallowedallowedallowedallowedallowed–
342curseforge.comUnreadableunknownunknownunknownunknownunknownunknownunknownunknown–
345warnerbros.comNo robots.txtallowedallowedallowedallowedallowedallowedallowedallowed–
352ted.comBlocks training onlyblockedpartialpartialblockedpartialpartialblockedblocked–
357disneyplus.comBlocks some AI searchpartialpartialpartialpartialpartialpartialpartialpartial–
358xbox.comOpenallowedallowedallowedallowedallowedallowedallowedallowed–
385chess.comOpenallowedallowedallowedallowedallowedallowedallowedallowed–
419youku.comNo robots.txtallowedallowedallowedallowedallowedallowedallowedallowed–
432playrix.comNo robots.txtallowedallowedallowedallowedallowedallowedallowedallowed–
433nexusmods.comUnreadableunknownunknownunknownunknownunknownunknownunknownunknown–
435ubi.comUnreadableunknownunknownunknownunknownunknownunknownunknownunknown–
437pixiv.netBlocks training onlyblockedallowedallowedblockedallowedallowedblockedblocked–
440flightradar24.comBlocks training onlyblockedallowedallowedallowedallowedallowedallowedblockedyes
447poki.comOpenallowedallowedallowedallowedallowedallowedallowedallowed–
492crazygames.comOpenallowedallowedallowedallowedallowedallowedallowedallowed–
495riotgames.comOpenallowedallowedallowedallowedallowedallowedallowedallowed–
496tabelog.comBlocks training onlyblockedallowedallowedallowedallowedallowedblockedallowed–
498aparat.comOpenallowedallowedallowedallowedallowedallowedallowedallowed–
504giphy.comOpenallowedallowedallowedallowedallowedallowedallowedallowed–
508cookpad.comBlocks some AI searchblockedallowedblockedblockedallowedblockedblockedblocked–
532tbs.co.jpNo robots.txtallowedallowedallowedallowedallowedallowedallowedallowed–
550nexon.comNo robots.txtallowedallowedallowedallowedallowedallowedallowedallowed–
560lichess.orgOpenallowedallowedallowedallowedallowedallowedallowedallowed–
562pandora.comBlocks training onlyblockedallowedallowedblockedallowedallowedblockedblocked–
572rutube.ruOpenallowedallowedallowedallowedallowedallowedallowedallowed–
604battle.netUnreadableunknownunknownunknownunknownunknownunknownunknownunknown–
618netkeiba.comNo robots.txtallowedallowedallowedallowedallowedallowedallowedallowed–
619playhop.comOpenallowedallowedallowedallowedallowedallowedallowedallowed–
624lookmovie2.toOpenallowedallowedallowedallowedallowedallowedallowedallowed–
641aternos.orgOpenallowedallowedallowedallowedallowedallowedallowedallowed–
652cardgames.ioOpenallowedallowedallowedallowedallowedallowedallowedallowed–
663mangabuff.ruUnreadableunknownunknownunknownunknownunknownunknownunknownunknown–
674filmweb.plOpenallowedallowedallowedallowedallowedallowedallowedallowed–
676zedge.netBlocks training onlyblockedallowedallowedallowedallowedallowedallowedallowed–
690gamewith.jpOpenallowedallowedallowedallowedallowedallowedallowedallowed–
700myinstants.comOpenallowedallowedallowedallowedallowedallowedallowedallowed–
725hanime1.meOpenallowedallowedallowedallowedallowedallowedallowedallowed–
729musicdel.irOpenallowedallowedallowedallowedallowedallowedallowedallowedyes
731playfabapi.comUnreadableunknownunknownunknownunknownunknownunknownunknownunknown–
745myreadingmanga.infoUnreadableunknownunknownunknownunknownunknownunknownunknownunknown–
762moe.videoOpenallowedallowedallowedallowedallowedallowedallowedallowedyes
767plex.tvOpenallowedallowedallowedallowedallowedallowedallowedallowed–
770kingbokep.tvOpenallowedallowedallowedallowedallowedallowedallowedallowed–
771nustargame.comOpenallowedallowedallowedallowedallowedallowedallowedallowedyes
784skipthegames.comOpenallowedallowedallowedallowedallowedallowedallowedallowed–
792arionplay.comOpenallowedallowedallowedallowedallowedallowedallowedallowed–
818yasyadong02.tvNo robots.txtallowedallowedallowedallowedallowedallowedallowedallowed–
820mangaraw.acOpenallowedallowedallowedallowedallowedallowedallowedallowed–
823playinboss.comNo robots.txtallowedallowedallowedallowedallowedallowedallowedallowed–
825kolkataff.tvOpenallowedallowedallowedallowedallowedallowedallowedallowed–
826filmyfly.soyUnreadableunknownunknownunknownunknownunknownunknownunknownunknown–
864pluto.tvOpenallowedallowedallowedallowedallowedallowedallowedallowed–
900hulu.comBlocks some AI searchpartialpartialpartialpartialpartialpartialpartialpartial–
907iqiyi.comBlocks some AI searchblockedallowedblockedblockedallowedblockedallowedallowed–
910gamebanana.comOpenallowedallowedallowedallowedallowedallowedallowedallowed–
999videoxx.comOpenallowedallowedallowedallowedallowedallowedallowedallowed–

FAQ

Do entertainment websites block AI crawlers?
As of 2026-10-01, 13.2% of the 68 entertainment websites whose robots.txt we could read block GPTBot, 8.8% block ClaudeBot and 5.9% block PerplexityBot (all sites: 17%, 16.3%, 12.7%). 7 of 77 publish an llms.txt file.
Which entertainment websites block GPTBot?
nintendo.com, ted.com, pixiv.net, flightradar24.com, tabelog.com, cookpad.com, pandora.com, zedge.net, iqiyi.com.

Other categories