AI Crawler Index · category
Do entertainment websites block AI crawlers?
As of 2026-11-16, 19.6% of the 332 entertainment websites whose robots.txt we could read block GPTBot, 18.7% block ClaudeBot and 4.2% block PerplexityBot (all sites: 15.4%, 14.7%, 8.2%). 30 of 370 publish an llms.txt file.
Block rate by crawler: entertainment vs all sites
| Crawler | Operator | Entertainment | All sites |
|---|---|---|---|
| Bytespider | ByteDance | 20.5% | 16.2% |
| CCBot | Common Crawl | 20.2% | 16.4% |
| GPTBot | OpenAI | 19.6% | 15.4% |
| Meta-ExternalAgent | Meta | 19% | 13.6% |
| cohere-ai | Cohere | 19% | 12.7% |
| ClaudeBot | Anthropic | 18.7% | 14.7% |
| Applebot-Extended | Apple | 18.4% | 13.4% |
| Amazonbot | Amazon | 17.5% | 13.2% |
| Google-Extended | 16.6% | 13.2% | |
| Google-CloudVertexBot | 13.9% | 8.8% | |
| MistralAI-Training | Mistral | 13.9% | 7.1% |
| ChatGPT-User | OpenAI | 4.5% | 7.4% |
| meta-webindexer | Meta | 4.5% | 5.1% |
| PerplexityBot | Perplexity | 4.2% | 8.2% |
| DuckAssistBot | DuckDuckGo | 3.9% | 5.9% |
| Perplexity-User | Perplexity | 3.6% | 6.1% |
| meta-externalfetcher | Meta | 3.6% | 6.5% |
| MistralAI-User | Mistral | 3.6% | 5.7% |
| Claude-User | Anthropic | 3.3% | 5.9% |
| Applebot | Apple | 3.3% | 3.2% |
| OAI-SearchBot | OpenAI | 3% | 5.8% |
| Claude-SearchBot | Anthropic | 2.7% | 5.7% |
| Amzn-SearchBot | Amazon | 2.7% | 4.4% |
| Amzn-User | Amazon | 2.4% | 4.4% |
| MistralAI-Index | Mistral | 2.4% | 3.6% |
| OAI-AdsBot | OpenAI | 1.8% | 3% |
| Google-GeminiNotebook | 1.8% | 3.3% | |
| meta-externalads | Meta | 1.8% | 2.9% |
| Google-Agent | 1.5% | 3.6% |
Top 150 of 370 entertainment sites in the index
✕ blocked · ◐ partially blocked (some paths) · ✓ allowed · ? robots.txt unreadable. "#" is the position in our popularity-ordered sample.
| # | Site | Policy | GPTBot | OAI-Search | ChatGPT-User | ClaudeBot | Claude-Search | Perplexity | Google-Ext. | CCBot | llms.txt |
|---|---|---|---|---|---|---|---|---|---|---|---|
| 5 | youtube.com | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 19 | netflix.com | Blocks some AI search | partial | partial | partial | partial | partial | partial | partial | blocked | – |
| 29 | roblox.com | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 31 | spotify.com | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 38 | vimeo.com | Blocks training only | partial | allowed | allowed | partial | allowed | allowed | partial | blocked | – |
| 52 | amazonvideo.com | Blocks some AI search | allowed | allowed | blocked | allowed | allowed | blocked | allowed | blocked | – |
| 53 | epicgames.com | Unreadable | unknown | unknown | unknown | unknown | unknown | unknown | unknown | unknown | – |
| 81 | soundcloud.com | Open | partial | partial | partial | partial | partial | partial | partial | partial | – |
| 97 | twitch.tv | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 128 | steampowered.com | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 146 | bilibili.com | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | yes |
| 152 | ea.com | Blocks training only | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | yes |
| 186 | dailymotion.com | Blocks training only | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | yes |
| 187 | steamcommunity.com | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 191 | supercell.com | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 217 | ivi.ru | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 265 | playstation.com | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | yes |
| 277 | nintendo.com | Blocks training only | blocked | allowed | allowed | blocked | allowed | allowed | blocked | blocked | – |
| 295 | primevideo.com | Blocks some AI search | allowed | allowed | blocked | allowed | allowed | blocked | allowed | blocked | – |
| 315 | kick.com | Blocks some AI search | blocked | allowed | allowed | blocked | allowed | allowed | blocked | blocked | – |
| 325 | itch.io | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 334 | imgur.com | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 347 | warnerbros.com | No robots.txt | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 349 | curseforge.com | Unreadable | unknown | unknown | unknown | unknown | unknown | unknown | unknown | unknown | – |
| 356 | ted.com | Blocks training only | blocked | partial | partial | blocked | partial | partial | blocked | blocked | – |
| 360 | disneyplus.com | Blocks some AI search | partial | partial | partial | partial | partial | partial | partial | partial | – |
| 362 | xbox.com | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 389 | chess.com | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 422 | youku.com | No robots.txt | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 436 | nexusmods.com | Unreadable | unknown | unknown | unknown | unknown | unknown | unknown | unknown | unknown | – |
| 438 | playrix.com | No robots.txt | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 441 | pixiv.net | Blocks training only | blocked | allowed | allowed | blocked | allowed | allowed | blocked | blocked | – |
| 443 | ubi.com | Unreadable | unknown | unknown | unknown | unknown | unknown | unknown | unknown | unknown | – |
| 446 | flightradar24.com | Blocks training only | blocked | allowed | allowed | allowed | allowed | allowed | allowed | blocked | yes |
| 448 | poki.com | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 499 | crazygames.com | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 501 | tabelog.com | Blocks training only | blocked | allowed | allowed | allowed | allowed | allowed | blocked | allowed | – |
| 506 | giphy.com | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 507 | aparat.com | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 508 | riotgames.com | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 517 | cookpad.com | Blocks some AI search | blocked | allowed | blocked | blocked | allowed | blocked | blocked | blocked | – |
| 527 | nexon.com | No robots.txt | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 542 | tbs.co.jp | No robots.txt | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 566 | lichess.org | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 570 | pandora.com | Blocks training only | blocked | allowed | allowed | blocked | allowed | allowed | blocked | blocked | – |
| 579 | rutube.ru | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 614 | battle.net | Unreadable | unknown | unknown | unknown | unknown | unknown | unknown | unknown | unknown | – |
| 626 | netkeiba.com | No robots.txt | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 629 | playhop.com | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 633 | lookmovie2.to | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 657 | aternos.org | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 663 | cardgames.io | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 674 | mangabuff.ru | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 679 | filmweb.pl | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 685 | zedge.net | Blocks training only | blocked | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 703 | gamewith.jp | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 707 | myinstants.com | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 739 | hanime1.me | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 741 | musicdel.ir | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | yes |
| 758 | myreadingmanga.info | Blocks some AI search | blocked | allowed | allowed | blocked | allowed | allowed | blocked | blocked | – |
| 779 | plex.tv | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 781 | nustargame.com | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | yes |
| 782 | kingbokep.tv | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 786 | playfabapi.com | Unreadable | unknown | unknown | unknown | unknown | unknown | unknown | unknown | unknown | – |
| 798 | skipthegames.com | Unreadable | unknown | unknown | unknown | unknown | unknown | unknown | unknown | unknown | – |
| 806 | arionplay.com | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | yes |
| 823 | moe.video | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | yes |
| 835 | yasyadong02.tv | No robots.txt | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 836 | mangaraw.ac | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 838 | playinboss.com | No robots.txt | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 840 | kolkataff.tv | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 841 | filmyfly.soy | No robots.txt | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 887 | pluto.tv | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 888 | gamebanana.com | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 925 | hulu.com | Unreadable | unknown | unknown | unknown | unknown | unknown | unknown | unknown | unknown | – |
| 935 | iqiyi.com | Blocks some AI search | blocked | allowed | blocked | blocked | allowed | blocked | allowed | allowed | – |
| 952 | videoxx.com | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 1041 | 5movierulz.kitchen | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 1044 | mp4moviez.rip | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 1045 | moviesdatamil.me | Unreadable | unknown | unknown | unknown | unknown | unknown | unknown | unknown | unknown | – |
| 1046 | moviezwap.taxi | Blocks some AI search | blocked | allowed | allowed | blocked | allowed | allowed | blocked | blocked | – |
| 1047 | filmywap.technology | Unreadable | unknown | unknown | unknown | unknown | unknown | unknown | unknown | unknown | – |
| 1048 | filmyzilla63.com | Blocks some AI search | blocked | allowed | allowed | blocked | allowed | allowed | blocked | blocked | – |
| 1049 | filmyzilla56.com | Blocks some AI search | blocked | allowed | allowed | blocked | allowed | allowed | blocked | blocked | – |
| 1050 | filmyzilla62.com | Blocks some AI search | blocked | allowed | allowed | blocked | allowed | allowed | blocked | blocked | – |
| 1092 | vkvideo.ru | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 1202 | zeasn.tv | No robots.txt | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 1213 | gameanalytics.com | Blocks training only | allowed | allowed | allowed | allowed | allowed | allowed | allowed | blocked | yes |
| 1218 | samba.tv | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 1331 | jwplayer.com | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 1335 | rockstargames.com | Blocks training only | blocked | allowed | allowed | blocked | allowed | allowed | allowed | blocked | – |
| 1358 | weplayapp.com | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | yes |
| 1371 | like.video | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 1418 | gamespot.com | Blocks some AI search | blocked | allowed | allowed | blocked | allowed | allowed | blocked | blocked | – |
| 1580 | medal.tv | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 1587 | v-videoapp.com | No robots.txt | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 1647 | bigolive.tv | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 1681 | nicovideo.jp | Blocks training only | blocked | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 1701 | myanimelist.net | Blocks some AI search | blocked | allowed | allowed | blocked | allowed | allowed | blocked | blocked | – |
| 1753 | videolan.org | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 1777 | ispot.tv | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 1835 | likeevideo.ru | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 2048 | centurygame.com | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 2076 | tcgplayer.com | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 2090 | likee.video | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 2095 | kingsgroupgames.com | No robots.txt | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 2229 | vkplay.ru | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 2239 | jennymovies.com | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 2242 | gameloft.com | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 2276 | okko.tv | Unreadable | unknown | unknown | unknown | unknown | unknown | unknown | unknown | unknown | – |
| 2309 | boardgamegeek.com | Unreadable | unknown | unknown | unknown | unknown | unknown | unknown | unknown | unknown | – |
| 2316 | farlightgames.com | No robots.txt | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | yes |
| 2350 | playfamily.ru | Unreadable | unknown | unknown | unknown | unknown | unknown | unknown | unknown | unknown | – |
| 2381 | movie-box.co | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 2418 | lectormangass.net | Blocks some AI search | blocked | allowed | allowed | blocked | allowed | allowed | blocked | blocked | – |
| 2426 | movieswood.cloud | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 2430 | filmyzilla52.com | Blocks some AI search | blocked | allowed | allowed | blocked | allowed | allowed | blocked | blocked | – |
| 2450 | boomplaymusic.com | Blocks training only | allowed | allowed | allowed | blocked | allowed | allowed | allowed | blocked | – |
| 2461 | my.games | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 2627 | ivideon.com | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 2670 | ustream.tv | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 2796 | arte.tv | Blocks all AI search | blocked | blocked | blocked | blocked | blocked | blocked | blocked | blocked | – |
| 2816 | pscp.tv | Blocks some AI search | blocked | allowed | allowed | blocked | allowed | allowed | blocked | blocked | – |
| 2901 | gamestop.com | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 2952 | gameinstaller.ru | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 2982 | videotron.ca | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 3034 | gamer.com.tw | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 3100 | stadtgame.com | Unreadable | unknown | unknown | unknown | unknown | unknown | unknown | unknown | unknown | – |
| 3110 | gamerant.com | Blocks training only | blocked | allowed | allowed | blocked | allowed | allowed | blocked | blocked | – |
| 3118 | filmaffinity.com | Unreadable | unknown | unknown | unknown | unknown | unknown | unknown | unknown | unknown | – |
| 3122 | adplay.ru | No robots.txt | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 3304 | pcgamer.com | Blocks some AI search | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 3309 | bhphotovideo.com | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 3316 | fubo.tv | No robots.txt | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 3320 | game8.co | Blocks training only | blocked | allowed | allowed | allowed | allowed | allowed | blocked | allowed | – |
| 3399 | movieweb.com | Blocks training only | blocked | allowed | allowed | blocked | allowed | allowed | blocked | blocked | – |
| 3411 | mangadex.org | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 3429 | playtika.com | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 3443 | playground.ru | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 3519 | devline.tv | No robots.txt | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 3524 | prodigygame.com | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | yes |
| 3546 | 9gag.com | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | yes |
| 3553 | ahagamecenter.com | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 3591 | nimo.tv | Blocks some AI search | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 3634 | raiplay.it | Blocks some AI search | blocked | allowed | blocked | blocked | allowed | blocked | blocked | blocked | – |
| 3708 | plaync.com | No robots.txt | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 3724 | videoamp.com | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 3733 | playdigo.com | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | yes |
| 3748 | microsoftcasualgames.com | No robots.txt | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 3751 | gamesradar.com | Blocks some AI search | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
Sorted by popularity. Every site has its own page: all 10,000 sites (A-Z lists), or download the CSV.
FAQ
- Do entertainment websites block AI crawlers?
- As of 2026-11-16, 19.6% of the 332 entertainment websites whose robots.txt we could read block GPTBot, 18.7% block ClaudeBot and 4.2% block PerplexityBot (all sites: 15.4%, 14.7%, 8.2%). 30 of 370 publish an llms.txt file.
- Which entertainment websites block GPTBot?
- nintendo.com, kick.com, ted.com, pixiv.net, flightradar24.com, tabelog.com, cookpad.com, pandora.com, zedge.net, myreadingmanga.info, and more (full list on this page).
Other categories
- AI GPTBot 10%
- E-commerce GPTBot 14.7%
- Education GPTBot 5.6%
- Finance GPTBot 6.9%
- Government GPTBot 3%
- News & media GPTBot 33.7%
- Other GPTBot 15.1%
- Reference GPTBot 27.9%
- Social GPTBot 37.3%
- Tech GPTBot 3.7%
- Telecom GPTBot 0%
- Travel GPTBot 15.4%