TidyToolsAI Crawler Index · data as of · updated daily

AI Crawler Index · category

Do social websites block AI crawlers?

As of 2026-10-01, 33.3% of the 48 social websites whose robots.txt we could read block GPTBot, 27.1% block ClaudeBot and 25% block PerplexityBot (all sites: 17%, 16.3%, 12.7%). 2 of 54 publish an llms.txt file.

Block rate by crawler: social vs all sites

CrawlerOperatorSocialAll sites
GPTBotOpenAI33.3%17%
Meta-ExternalAgentMeta29.2%14.4%
CCBotCommon Crawl29.2%19.5%
ClaudeBotAnthropic27.1%16.3%
Google-ExtendedGoogle27.1%14%
Applebot-ExtendedApple27.1%14.3%
BytespiderByteDance27.1%17.7%
PerplexityBotPerplexity25%12.7%
ChatGPT-UserOpenAI22.9%9.6%
AmazonbotAmazon22.9%13.1%
cohere-aiCohere22.9%13.7%
Claude-UserAnthropic20.8%8.3%
Perplexity-UserPerplexity20.8%8.9%
Google-CloudVertexBotGoogle20.8%7.2%
MistralAI-UserMistral20.8%8.4%
OAI-SearchBotOpenAI18.8%6.7%
OAI-AdsBotOpenAI18.8%4.3%
Claude-SearchBotAnthropic18.8%8.1%
Google-GeminiNotebookGoogle18.8%4.3%
Google-AgentGoogle18.8%4.5%
meta-webindexerMeta18.8%6.3%
meta-externalfetcherMeta18.8%8.7%
Amzn-SearchBotAmazon18.8%5.9%
Amzn-UserAmazon18.8%5.5%
DuckAssistBotDuckDuckGo18.8%8.1%
MistralAI-IndexMistral18.8%4.7%
MistralAI-TrainingMistral18.8%4.7%
meta-externaladsMeta16.7%3.8%
ApplebotApple2.1%2.6%

All 54 social sites in the index

✕ blocked · ◐ partially blocked (some paths) · ✓ allowed · ? robots.txt unreadable. "#" is the position in our popularity-ordered sample.

#SitePolicyGPTBotOAI-SearchChatGPT-UserClaudeBotClaude-SearchPerplexityGoogle-Ext.CCBotllms.txt
3facebook.comBlocks some AI searchpartialblockedblockedpartialblockedpartialpartialblocked–
7instagram.comBlocks some AI searchblockedblockedblockedblockedblockedblockedblockedblocked–
9twitter.comBlocks some AI searchblockedblockedblockedblockedblockedblockedblockedblocked–
10linkedin.comBlocks some AI searchblockedpartialblockedblockedpartialblockedblockedblocked–
21x.comBlocks some AI searchblockedblockedblockedblockedblockedblockedblockedblocked–
22skype.comNo robots.txtallowedallowedallowedallowedallowedallowedallowedallowed–
24pinterest.comBlocks some AI searchblockedblockedblockedblockedblockedblockedblockedblocked–
25tiktok.comBlocks some AI searchblockedblockedblockedblockedblockedblockedblockedblocked–
26whatsapp.comBlocks some AI searchblockedallowedallowedblockedallowedblockedblockedallowed–
44t.meNo robots.txtallowedallowedallowedallowedallowedallowedallowedallowed–
46reddit.comBlocks all AI searchblockedblockedblockedblockedblockedblockedblockedblocked–
52snapchat.comBlocks some AI searchblockedpartialpartialblockedpartialpartialblockedblocked–
56vk.comOpenallowedallowedallowedallowedallowedallowedallowedallowed–
60discord.ggOpenallowedallowedallowedallowedallowedallowedallowedallowed–
64tumblr.comUnreadableunknownunknownunknownunknownunknownunknownunknownunknown–
72vk.ruOpenallowedallowedallowedallowedallowedallowedallowedallowed–
103telegram.orgNo robots.txtallowedallowedallowedallowedallowedallowedallowedallowed–
127weibo.comBlocks some AI searchblockedblockedblockedblockedblockedblockedblockedblocked–
170bsky.appOpenallowedallowedallowedallowedallowedallowedallowedallowed–
178line.meOpenallowedallowedallowedallowedallowedallowedallowedallowed–
220behance.netBlocks training onlyblockedallowedallowedblockedallowedallowedblockedblocked–
234gotinder.comOpenallowedallowedallowedallowedallowedallowedallowedallowed–
250deviantart.comOpenallowedallowedallowedallowedallowedallowedallowedallowed–
263threads.comBlocks some AI searchblockedblockedblockedblockedblockedblockedblockedblocked–
305viber.comOpenallowedallowedallowedallowedallowedallowedallowedallowed–
337patreon.comBlocks some AI searchallowedallowedallowedallowedallowedblockedallowedallowed–
344meta.comOpenallowedallowedallowedallowedallowedallowedallowedallowed–
348wattpad.comBlocks training onlyblockedallowedallowedallowedallowedallowedallowedallowed–
353ameblo.jpOpenallowedallowedallowedallowedallowedallowedallowedallowed–
380livejournal.comOpenallowedallowedallowedallowedallowedallowedallowedallowed–
390pikabu.ruOpenallowedallowedallowedallowedallowedallowedallowedallowed–
396myspace.comOpenallowedallowedallowedallowedallowedallowedallowedallowed–
423note.comOpenallowedallowedallowedallowedallowedallowedallowedallowed–
430change.orgOpenallowedallowedallowedallowedallowedallowedallowedallowed–
454xing.comOpenpartialallowedallowedpartialallowedpartialallowedallowedyes
480ameba.jpOpenallowedallowedallowedallowedallowedallowedallowedallowed–
484archiveofourown.orgBlocks some AI searchblockedallowedblockedallowedallowedallowedallowedblocked–
507gofundme.comOpenallowedallowedallowedallowedallowedallowedallowedallowed–
515kickstarter.comUnreadableunknownunknownunknownunknownunknownunknownunknownunknown–
545fmkorea.comOpenpartialallowedallowedpartialallowedpartialpartialpartial–
548dcinside.comBlocks some AI searchblockedallowedallowedblockedallowedblockedblockedblocked–
579livedoor.comOpenallowedallowedallowedallowedallowedallowedallowedallowed–
589dribbble.comOpenallowedallowedallowedallowedallowedallowedallowedallowed–
591fanfiction.netUnreadableunknownunknownunknownunknownunknownunknownunknownunknown–
595douyin.comOpenpartialpartialpartialallowedallowedpartialpartialallowed–
608arca.liveUnreadableunknownunknownunknownunknownunknownunknownunknownunknown–
609ruliweb.comUnreadableunknownunknownunknownunknownunknownunknownunknownunknown–
662syosetu.comBlocks training onlyallowedallowedallowedallowedallowedallowedallowedallowed–
665ficbook.netOpenallowedallowedallowedallowedallowedallowedallowedallowed–
693badoo.comOpenallowedallowedallowedallowedallowedallowedallowedallowed–
710ptt.ccUnreadableunknownunknownunknownunknownunknownunknownunknownunknown–
714ninisite.comOpenallowedallowedallowedallowedallowedallowedallowedallowed–
717dogdrip.netOpenallowedallowedallowedallowedallowedallowedallowedallowed–
952meetup.comBlocks training onlyblockedallowedallowedallowedallowedallowedallowedallowedyes

FAQ

Do social websites block AI crawlers?
As of 2026-10-01, 33.3% of the 48 social websites whose robots.txt we could read block GPTBot, 27.1% block ClaudeBot and 25% block PerplexityBot (all sites: 17%, 16.3%, 12.7%). 2 of 54 publish an llms.txt file.
Which social websites block GPTBot?
instagram.com, twitter.com, linkedin.com, x.com, pinterest.com, tiktok.com, whatsapp.com, reddit.com, snapchat.com, weibo.com, and more (full list on this page).

Other categories