TidyToolsAI Crawler Index · data as of · updated daily

AI Crawler Index · category

Do social websites block AI crawlers?

As of 2026-11-16, 37.3% of the 51 social websites whose robots.txt we could read block GPTBot, 33.3% block ClaudeBot and 27.5% block PerplexityBot (all sites: 15.4%, 14.7%, 8.2%). 2 of 56 publish an llms.txt file.

Block rate by crawler: social vs all sites

CrawlerOperatorSocialAll sites
GPTBotOpenAI37.3%15.4%
ClaudeBotAnthropic33.3%14.7%
Google-ExtendedGoogle33.3%13.2%
Applebot-ExtendedApple33.3%13.4%
Meta-ExternalAgentMeta33.3%13.6%
CCBotCommon Crawl33.3%16.4%
AmazonbotAmazon29.4%13.2%
BytespiderByteDance29.4%16.2%
PerplexityBotPerplexity27.5%8.2%
cohere-aiCohere25.5%12.7%
ChatGPT-UserOpenAI23.5%7.4%
Google-CloudVertexBotGoogle23.5%8.8%
Claude-UserAnthropic21.6%5.9%
Perplexity-UserPerplexity21.6%6.1%
meta-externalfetcherMeta21.6%6.5%
MistralAI-UserMistral21.6%5.7%
MistralAI-TrainingMistral21.6%7.1%
OAI-SearchBotOpenAI19.6%5.8%
OAI-AdsBotOpenAI19.6%3%
Claude-SearchBotAnthropic19.6%5.7%
Google-GeminiNotebookGoogle19.6%3.3%
Google-AgentGoogle19.6%3.6%
meta-webindexerMeta19.6%5.1%
Amzn-SearchBotAmazon19.6%4.4%
Amzn-UserAmazon19.6%4.4%
DuckAssistBotDuckDuckGo19.6%5.9%
MistralAI-IndexMistral19.6%3.6%
meta-externaladsMeta17.6%2.9%
ApplebotApple2%3.2%

All 56 social sites in the index

✕ blocked · ◐ partially blocked (some paths) · ✓ allowed · ? robots.txt unreadable. "#" is the position in our popularity-ordered sample.

#SitePolicyGPTBotOAI-SearchChatGPT-UserClaudeBotClaude-SearchPerplexityGoogle-Ext.CCBotllms.txt
3facebook.comBlocks some AI searchblockedblockedblockedblockedblockedblockedblockedblocked–
7instagram.comBlocks some AI searchblockedblockedblockedblockedblockedblockedblockedblocked–
9twitter.comBlocks some AI searchblockedblockedblockedblockedblockedblockedblockedblocked–
11linkedin.comBlocks some AI searchblockedpartialblockedblockedpartialblockedblockedblocked–
21x.comBlocks some AI searchblockedblockedblockedblockedblockedblockedblockedblocked–
23skype.comNo robots.txtallowedallowedallowedallowedallowedallowedallowedallowed–
25pinterest.comBlocks some AI searchblockedblockedblockedblockedblockedblockedblockedblocked–
26tiktok.comBlocks some AI searchblockedblockedblockedblockedblockedblockedblockedblocked–
27whatsapp.comBlocks some AI searchblockedallowedallowedblockedallowedblockedblockedallowed–
44t.meNo robots.txtallowedallowedallowedallowedallowedallowedallowedallowed–
48reddit.comBlocks all AI searchblockedblockedblockedblockedblockedblockedblockedblocked–
55snapchat.comBlocks some AI searchblockedpartialpartialblockedpartialpartialblockedblocked–
59vk.comOpenallowedallowedallowedallowedallowedallowedallowedallowed–
61discord.ggOpenallowedallowedallowedallowedallowedallowedallowedallowed–
66tumblr.comBlocks training onlyallowedallowedallowedblockedallowedallowedblockedblocked–
74vk.ruOpenallowedallowedallowedallowedallowedallowedallowedallowed–
103telegram.orgNo robots.txtallowedallowedallowedallowedallowedallowedallowedallowed–
129weibo.comBlocks some AI searchblockedblockedblockedblockedblockedblockedblockedblocked–
131ok.ruOpenallowedallowedallowedallowedallowedallowedallowedallowed–
173bsky.appOpenallowedallowedallowedallowedallowedallowedallowedallowed–
180line.meOpenallowedallowedallowedallowedallowedallowedallowedallowed–
225behance.netBlocks training onlyblockedallowedallowedblockedallowedallowedblockedblocked–
229gotinder.comOpenallowedallowedallowedallowedallowedallowedallowedallowed–
253deviantart.comOpenallowedallowedallowedallowedallowedallowedallowedallowed–
264threads.comBlocks some AI searchblockedblockedblockedblockedblockedblockedblockedblocked–
312viber.comOpenallowedallowedallowedallowedallowedallowedallowedallowed–
343meta.comOpenallowedallowedallowedallowedallowedallowedallowedallowed–
352wattpad.comBlocks training onlyblockedallowedallowedallowedallowedallowedallowedallowed–
353patreon.comBlocks some AI searchallowedallowedallowedallowedallowedblockedallowedallowed–
358ameblo.jpOpenallowedallowedallowedallowedallowedallowedallowedallowed–
383livejournal.comUnreadableunknownunknownunknownunknownunknownunknownunknownunknown–
393pikabu.ruUnreadableunknownunknownunknownunknownunknownunknownunknownunknown–
398myspace.comOpenallowedallowedallowedallowedallowedallowedallowedallowed–
426note.comOpenallowedallowedallowedallowedallowedallowedallowedallowed–
434change.orgOpenallowedallowedallowedallowedallowedallowedallowedallowed–
456xing.comOpenpartialallowedallowedpartialpartialpartialallowedallowedyes
486ameba.jpOpenallowedallowedallowedallowedallowedallowedallowedallowed–
489archiveofourown.orgBlocks some AI searchblockedallowedblockedallowedallowedallowedallowedblocked–
513gofundme.comOpenallowedallowedallowedallowedallowedallowedallowedallowed–
525kickstarter.comUnreadableunknownunknownunknownunknownunknownunknownunknownunknown–
555fmkorea.comOpenpartialallowedallowedpartialallowedpartialpartialpartial–
557dcinside.comBlocks some AI searchblockedallowedallowedblockedallowedblockedblockedblocked–
580douyin.comOpenpartialpartialpartialallowedallowedpartialpartialallowed–
589livedoor.comOpenallowedallowedallowedallowedallowedallowedallowedallowed–
599fanfiction.netUnreadableunknownunknownunknownunknownunknownunknownunknownunknown–
601dribbble.comOpenallowedallowedallowedallowedallowedallowedallowedallowed–
617ruliweb.comOpenallowedallowedallowedallowedallowedallowedallowedallowed–
619arca.liveOpenallowedallowedallowedallowedallowedallowedallowedallowed–
671syosetu.comBlocks training onlyallowedallowedallowedallowedallowedallowedallowedallowed–
672ficbook.netOpenallowedallowedallowedallowedallowedallowedallowedallowed–
718ptt.ccUnreadableunknownunknownunknownunknownunknownunknownunknownunknown–
721ninisite.comOpenallowedallowedallowedallowedallowedallowedallowedallowed–
730dogdrip.netBlocks some AI searchblockedallowedallowedblockedallowedallowedblockedblocked–
821badoo.comOpenallowedallowedallowedallowedallowedallowedallowedallowed–
994meetup.comBlocks training onlyblockedallowedallowedallowedallowedallowedallowedallowedyes
1431nextdoor.comBlocks some AI searchblockedblockedblockedblockedblockedblockedblockedblocked–

FAQ

Do social websites block AI crawlers?
As of 2026-11-16, 37.3% of the 51 social websites whose robots.txt we could read block GPTBot, 33.3% block ClaudeBot and 27.5% block PerplexityBot (all sites: 15.4%, 14.7%, 8.2%). 2 of 56 publish an llms.txt file.
Which social websites block GPTBot?
facebook.com, instagram.com, twitter.com, linkedin.com, x.com, pinterest.com, tiktok.com, whatsapp.com, reddit.com, snapchat.com, and more (full list on this page).

Other categories