TidyToolsData as of · updated daily

AI Crawler Index

Who blocks AI crawlers?

About 1 in 6 popular websites block OpenAI's GPTBot in robots.txt. We check the robots.txt of 1,005 of the world's most visited websites against 26 AI crawlers every day: training bots, AI search bots and the fetchers that run when a user asks a question.

17.3%
block GPTBot
top 100: 22%
16.5%
block ClaudeBot
top 100: 23.2%
6.9%
block OAI-SearchBot
ChatGPT search
49.2%
of news sites block ClaudeBot
n=126

Share of the 855 sites whose robots.txt we could read. "Block" means the home page is disallowed for that crawler.

Key findings

Block rate by crawler

CCBotCommon Crawl
19.5%
BytespiderByteDance
17.8%
GPTBotOpenAI
17.3%
ClaudeBotAnthropic
16.5%
Applebot-ExtendedApple
14.5%
Meta-ExternalAgentMeta
14.4%
Google-ExtendedGoogle
14.3%
cohere-aiCohere
13.9%
AmazonbotAmazon
13.3%
PerplexityBotPerplexity
13%
ChatGPT-UserOpenAI
9.9%
Perplexity-UserPerplexity
9.1%
meta-externalfetcherMeta
8.9%
MistralAI-UserMistral
8.7%
Claude-UserAnthropic
8.4%
DuckAssistBotDuckDuckGo
8.3%
Claude-SearchBotAnthropic
8.2%
OAI-SearchBotOpenAI
6.9%
meta-webindexerMeta
6.5%
Amzn-SearchBotAmazon
6.1%
Amzn-UserAmazon
5.7%
MistralAI-IndexMistral
4.9%
MistralAI-TrainingMistral
4.9%
Google-AgentGoogle
4.7%
Google-GeminiNotebookGoogle
4.6%
ApplebotApple
2.8%

By category

Share of readable sites in each category that block the crawler on the home page. Sorted by average block rate.

Category nGPTBotOAI-SearchChatGPT-UserClaudeBotClaude-SearchPerplexityGoogle-Ext.CCBotllms.txt
News & media 12643.7%21.4%28.6%49.2%27.8%45.2%41.3%58.7%10
Social 4833.3%18.8%22.9%27.1%18.8%25%27.1%29.2%2
Reference 4327.9%7%11.6%23.3%7%9.3%16.3%27.9%1
AI 911.1%11.1%11.1%0%0%11.1%22.2%11.1%1
E-commerce 5911.9%5.1%5.1%13.6%6.8%8.5%10.2%15.3%10
Other 26113.8%5.4%8%10.7%5.7%8.4%10.3%13.4%49
Entertainment 6814.7%1.5%7.4%10.3%1.5%7.4%10.3%14.7%7
Travel 1315.4%0%0%23.1%7.7%0%7.7%7.7%1
Government 336.1%0%3%6.1%0%3%3%6.1%1
Tech 1544.5%0.6%1.3%5.2%1.3%2.6%3.9%5.8%55
Education 200%0%0%0%0%0%0%0%0
Finance 160%0%0%0%0%0%0%0%3
Telecom 50%0%0%0%0%0%0%0%2

Policy mix

Changes since yesterday

This is the first day of the index (2026-09-30), so there is nothing to compare yet. From the next daily run, every site that starts or stops blocking an AI crawler shows up here.

Block rate over time

The daily series starts on 2026-09-30. A trend chart appears here once there are two days of data.

The top 100 sites

✕ blocked · ◐ partially blocked (some paths) · ✓ allowed · ? robots.txt unreadable. "#" is the position in our sample, ordered by popularity.

#SiteCategoryPolicyGPTBotOAI-SearchChatGPT-UserClaudeBotClaude-SearchPerplexityGoogle-Ext.CCBotllms.txt
1google.comTechOpenallowedallowedallowedallowedallowedallowedallowedallowed–
2cloudflare.comTechOpenallowedallowedallowedallowedallowedallowedallowedallowedyes
3facebook.comSocialBlocks some AI searchpartialblockedblockedpartialblockedpartialpartialblocked–
4microsoft.comTechOpenallowedallowedallowedallowedallowedallowedallowedallowed–
5youtube.comEntertainmentOpenallowedallowedallowedallowedallowedallowedallowedallowed–
6apple.comTechOpenallowedallowedallowedallowedallowedallowedallowedallowed–
7instagram.comSocialBlocks some AI searchblockedblockedblockedblockedblockedblockedblockedblocked–
8mail.ruTechOpenallowedallowedallowedallowedallowedallowedallowedallowed–
9twitter.comSocialBlocks some AI searchblockedblockedblockedblockedblockedblockedblockedblocked–
10linkedin.comSocialBlocks some AI searchblockedpartialblockedblockedpartialblockedblockedblocked–
11office.comTechNo robots.txtallowedallowedallowedallowedallowedallowedallowedallowed–
12amazon.comE-commerceBlocks some AI searchblockedblockedblockedblockedblockedblockedblockedblocked–
13azure.comOtherNo robots.txtallowedallowedallowedallowedallowedallowedallowedallowed–
14github.comTechBlocks training onlypartialpartialallowedpartialallowedpartialallowedallowedyes
15wikipedia.orgReferenceOpenallowedallowedallowedallowedallowedallowedallowedallowed–
16bing.comTechOpenallowedallowedallowedallowedallowedallowedallowedallowed–
17cloud.microsoftOtherNo robots.txtallowedallowedallowedallowedallowedallowedallowedallowed–
18netflix.comEntertainmentUnreadableunknownunknownunknownunknownunknownunknownunknownunknown–
19gandi.netOtherOpenallowedallowedallowedallowedallowedallowedallowedallowed–
20wordpress.orgTechOpenallowedallowedallowedallowedallowedallowedallowedallowedyes
21x.comSocialBlocks some AI searchblockedblockedblockedblockedblockedblockedblockedblocked–
22skype.comSocialNo robots.txtallowedallowedallowedallowedallowedallowedallowedallowed–
23icloud.comTechOpenallowedallowedallowedallowedallowedallowedallowedallowed–
24pinterest.comSocialBlocks some AI searchblockedblockedblockedblockedblockedblockedblockedblocked–
25tiktok.comSocialBlocks some AI searchblockedblockedblockedblockedblockedblockedblockedblocked–
26whatsapp.comSocialBlocks some AI searchblockedallowedallowedblockedallowedblockedblockedallowed–
27yahoo.comTechBlocks some AI searchblockedpartialblockedblockedpartialblockedblockedblocked–
28roblox.comEntertainmentOpenallowedallowedallowedallowedallowedallowedallowedallowed–
29spotify.comEntertainmentOpenallowedallowedallowedallowedallowedallowedallowedallowed–
30adobe.comTechOpenallowedallowedallowedallowedallowedallowedallowedallowedyes
31qq.comTechOpenallowedallowedallowedallowedallowedallowedallowedallowed–
32zoom.usTechOpenallowedallowedallowedallowedallowedallowedallowedallowed–
33nginx.orgTechOpenallowedallowedallowedallowedallowedallowedallowedallowed–
34baidu.comTechUnreadableunknownunknownunknownunknownunknownunknownunknownunknown–
35chatgpt.comAIBlocks some AI searchallowedallowedallowedallowedallowedblockedblockedblocked–
36vimeo.comEntertainmentBlocks training onlypartialallowedallowedpartialallowedallowedpartialblocked–
37openai.comAIOpenallowedallowedallowedallowedallowedallowedallowedallowed–
38opera.comTechOpenallowedallowedallowedallowedallowedallowedallowedallowedyes
39f5.comTechOpenallowedallowedallowedallowedallowedallowedallowedallowed–
40sentry.ioTechOpenallowedallowedallowedallowedallowedallowedallowedallowedyes
41windows.comOtherNo robots.txtallowedallowedallowedallowedallowedallowedallowedallowed–
42samsung.comTechOpenpartialpartialpartialallowedallowedpartialpartialallowedyes
43mozilla.orgTechOpenallowedallowedallowedallowedallowedallowedallowedallowed–
44t.meSocialNo robots.txtallowedallowedallowedallowedallowedallowedallowedallowed–
45dellemc.comOtherUnreadableunknownunknownunknownunknownunknownunknownunknownunknown–
46reddit.comSocialBlocks all AI searchblockedblockedblockedblockedblockedblockedblockedblocked–
47ui.comTechOpenallowedallowedallowedallowedallowedallowedallowedallowed–
48office365.comTechUnreadableunknownunknownunknownunknownunknownunknownunknownunknown–
49europa.euGovernmentOpenallowedallowedallowedallowedallowedallowedallowedallowed–
50amazonvideo.comEntertainmentBlocks some AI searchallowedallowedblockedallowedallowedblockedallowedblocked–
51epicgames.comEntertainmentUnreadableunknownunknownunknownunknownunknownunknownunknownunknown–
52snapchat.comSocialBlocks some AI searchblockedpartialpartialblockedpartialpartialblockedblocked–
53apache.orgTechOpenallowedallowedallowedallowedallowedallowedallowedallowed–
54reg.ruOtherNo robots.txtallowedallowedallowedallowedallowedallowedallowedallowed–
55dropbox.comTechOpenallowedallowedallowedallowedallowedallowedallowedallowedyes
56vk.comSocialOpenallowedallowedallowedallowedallowedallowedallowedallowed–
57intuit.comFinanceOpenallowedallowedallowedallowedallowedallowedallowedallowedyes
58unity3d.comTechBlocks training onlyallowedallowedallowedallowedallowedallowedallowedallowedyes
59nih.govGovernmentOpenallowedallowedallowedallowedallowedallowedallowedallowed–
60discord.ggSocialOpenallowedallowedallowedallowedallowedallowedallowedallowed–
61kaspersky.comTechOpenallowedallowedallowedallowedallowedallowedallowedallowedyes
62archive.orgReferenceOpenallowedallowedallowedallowedallowedallowedallowedallowed–
63godaddy.comTechUnreadableunknownunknownunknownunknownunknownunknownunknownunknown–
64tumblr.comSocialUnreadableunknownunknownunknownunknownunknownunknownunknownunknown–
65nytimes.comNews & mediaBlocks some AI searchblockedblockedblockedblockedblockedblockedblockedblocked–
66xiaomi.comTechUnreadableunknownunknownunknownunknownunknownunknownunknownunknown–
67mts.ruTelecomOpenallowedallowedallowedallowedallowedallowedallowedallowed–
68webex.comTechUnreadableunknownunknownunknownunknownunknownunknownunknownunknownyes
69paypal.comFinanceOpenallowedallowedallowedallowedallowedallowedallowedallowedyes
70spectrum.comTelecomUnreadableunknownunknownunknownunknownunknownunknownunknownunknown–
71shopify.comE-commerceOpenallowedallowedallowedallowedallowedallowedallowedallowedyes
72vk.ruSocialOpenallowedallowedallowedallowedallowedallowedallowedallowed–
73base.orgOtherOpenallowedallowedallowedallowedallowedallowedallowedallowedyes
74hosting24.comOtherUnreadableunknownunknownunknownunknownunknownunknownunknownunknown–
75macromedia.comOtherUnreadableunknownunknownunknownunknownunknownunknownunknownunknown–
76paytmfirstgames.comFinanceUnreadableunknownunknownunknownunknownunknownunknownunknownunknown–
77w3.orgReferenceOpenallowedallowedallowedallowedallowedallowedallowedallowed–
78flickr.comOtherBlocks some AI searchblockedpartialpartialblockedpartialpartialblockedblocked–
79miit.gov.cnGovernmentUnreadableunknownunknownunknownunknownunknownunknownunknownunknown–
80soundcloud.comEntertainmentOpenpartialpartialpartialpartialallowedpartialpartialpartial–
81telekom.netTelecomNo robots.txtallowedallowedallowedallowedallowedallowedallowedallowed–
82creativecommons.orgOtherOpenallowedallowedallowedallowedallowedallowedallowedallowed–
83medium.comNews & mediaBlocks some AI searchblockedallowedallowedblockedallowedallowedallowedallowed–
84cnn.comNews & mediaUnreadableunknownunknownunknownunknownunknownunknownunknownunknown–
85theguardian.comNews & mediaBlocks some AI searchallowedallowedallowedblockedblockedblockedallowedblocked–
86ebay.comE-commerceBlocks some AI searchblockedpartialpartialblockedpartialblockedallowedblocked–
87t-online.deNews & mediaBlocks some AI searchblockedallowedallowedblockedallowedblockedpartialblocked–
88oracle.comTechOpenpartialpartialpartialpartialallowedpartialpartialallowed–
89doi.orgReferenceNo robots.txtallowedallowedallowedallowedallowedallowedallowedallowed–
90android.comTechOpenpartialpartialallowedpartialpartialpartialpartialpartial–
91forbes.comNews & mediaBlocks some AI searchblockedallowedallowedblockedallowedblockedallowedblocked–
92bbc.comNews & mediaBlocks some AI searchblockedblockedblockedblockedallowedblockedblockedblocked–
93ubuntu.comTechOpenpartialpartialpartialpartialpartialpartialpartialallowedyes
94sciencedirect.comReferenceUnreadableunknownunknownunknownunknownunknownunknownunknownunknown–
95twitch.tvEntertainmentOpenallowedallowedallowedallowedallowedallowedallowedallowed–
96researchgate.netReferenceOpenallowedallowedallowedallowedallowedallowedallowedallowed–
97mit.eduEducationOpenallowedallowedallowedallowedallowedallowedallowedallowed–
98ozon.ruE-commerceUnreadableunknownunknownunknownunknownunknownunknownunknownunknown–
99cisco.comTechOpenallowedallowedallowedallowedallowedallowedallowedallowedyes
100comcast.netTelecomUnreadableunknownunknownunknownunknownunknownunknownunknownunknown–

Download the data

Free to reuse under CC BY 4.0. Please cite: TidyTools AI Crawler Index, 2026-09-30, https://tools.yukai.uk/ai-crawler-index

Methodology & caveats