TidyToolsAI Crawler Index · data as of · updated daily

AI Crawler Index · category

Do telecom websites block AI crawlers?

As of 2026-11-16, 0% of the 6 telecom websites whose robots.txt we could read block GPTBot, 0% block ClaudeBot and 0% block PerplexityBot (all sites: 15.4%, 14.7%, 8.2%). 2 of 12 publish an llms.txt file.

Block rate by crawler: telecom vs all sites

CrawlerOperatorTelecomAll sites
GPTBotOpenAI0%15.4%
OAI-SearchBotOpenAI0%5.8%
ChatGPT-UserOpenAI0%7.4%
OAI-AdsBotOpenAI0%3%
ClaudeBotAnthropic0%14.7%
Claude-SearchBotAnthropic0%5.7%
Claude-UserAnthropic0%5.9%
PerplexityBotPerplexity0%8.2%
Perplexity-UserPerplexity0%6.1%
Google-ExtendedGoogle0%13.2%
Google-CloudVertexBotGoogle0%8.8%
Google-GeminiNotebookGoogle0%3.3%
Google-AgentGoogle0%3.6%
ApplebotApple0%3.2%
Applebot-ExtendedApple0%13.4%
Meta-ExternalAgentMeta0%13.6%
meta-webindexerMeta0%5.1%
meta-externalfetcherMeta0%6.5%
meta-externaladsMeta0%2.9%
AmazonbotAmazon0%13.2%
Amzn-SearchBotAmazon0%4.4%
Amzn-UserAmazon0%4.4%
DuckAssistBotDuckDuckGo0%5.9%
MistralAI-UserMistral0%5.7%
MistralAI-IndexMistral0%3.6%
MistralAI-TrainingMistral0%7.1%
CCBotCommon Crawl0%16.4%
BytespiderByteDance0%16.2%
cohere-aiCohere0%12.7%

All 12 telecom sites in the index

✕ blocked · ◐ partially blocked (some paths) · ✓ allowed · ? robots.txt unreadable. "#" is the position in our popularity-ordered sample.

#SitePolicyGPTBotOAI-SearchChatGPT-UserClaudeBotClaude-SearchPerplexityGoogle-Ext.CCBotllms.txt
69mts.ruOpenallowedallowedallowedallowedallowedallowedallowedallowed–
73spectrum.comUnreadableunknownunknownunknownunknownunknownunknownunknownunknown–
89telekom.netNo robots.txtallowedallowedallowedallowedallowedallowedallowedallowed–
102comcast.netUnreadableunknownunknownunknownunknownunknownunknownunknownunknown–
158twc.comUnreadableunknownunknownunknownunknownunknownunknownunknownunknown–
361rt.ruUnreadableunknownunknownunknownunknownunknownunknownunknownunknown–
379att.comOpenallowedallowedallowedallowedallowedallowedallowedallowedyes
467docomo.ne.jpNo robots.txtallowedallowedallowedallowedallowedallowedallowedallowed–
502t-mobile.comUnreadableunknownunknownunknownunknownunknownunknownunknownunknown–
725ioh.co.idUnreadableunknownunknownunknownunknownunknownunknownunknownunknown–
852xfinity.comOpenallowedallowedallowedallowedallowedallowedallowedallowedyes
1132verizon.comOpenallowedallowedallowedallowedallowedallowedallowedallowed–

FAQ

Do telecom websites block AI crawlers?
As of 2026-11-16, 0% of the 6 telecom websites whose robots.txt we could read block GPTBot, 0% block ClaudeBot and 0% block PerplexityBot (all sites: 15.4%, 14.7%, 8.2%). 2 of 12 publish an llms.txt file.
Which telecom websites block GPTBot?
None in the index.

Other categories