TidyToolsAI Crawler Index · data as of · updated daily

AI crawler · Anthropic · Training

Which websites block ClaudeBot?

As of 2026-11-16, 1,221 of the 8,313 popular websites whose robots.txt we could read (14.7%) block ClaudeBot, Anthropic's AI training crawler, on the home page; 228 more block it on some paths. Among the top 100 sites: 25.3%.

14.7%
of 8,313 readable sites block ClaudeBot
25.3%
of the top 100
#4
most blocked of 29 AI crawlers
6,864
sites allow it on the home page

Blocked by category

CategorySites readBlock ClaudeBot
News & media39235.7%
Social5133.3%
Reference4323.3%
Travel1323.1%
Entertainment33218.7%
E-commerce11617.2%
Other665114.1%
Finance1026.9%
Tech1625.6%
Education2324.7%
Government2033%
AI100%
Telecom60%

Popular sites that block ClaudeBot (1,221)

Showing the 200 most popular of 1,221. The full list is in sites.csv.

Sites that block ClaudeBot on some paths (228)

Showing the 60 most popular of 228. The full list is in sites.csv.

Trend

DateShare blocking ClaudeBot
2026-11-1614.7%
2026-11-1514.6%
2026-11-1414.6%
2026-11-1314.6%
2026-11-1214.6%
2026-11-1114.6%
2026-11-1014.6%
2026-11-0914.7%
2026-11-0814.6%
2026-11-0714.7%
2026-11-0614.7%
2026-11-0514.7%
2026-11-0414.7%
2026-11-0314.6%
2026-11-0214.5%
2026-11-0114.5%
2026-10-3114.6%
2026-10-3014.5%
2026-10-2914.5%
2026-10-2813.9%
2026-10-2713.2%
2026-10-2612.6%
2026-10-2512.6%
2026-10-2412.5%
2026-10-2312.4%
2026-10-2212.4%
2026-10-2112.6%
2026-10-2011.9%
2026-10-1911.7%
2026-10-1811.6%

Latest changes

Block or allow ClaudeBot

User-agent: ClaudeBot
Disallow: /

Replace Disallow: / with Allow: / to let it in. A site that blocks training crawlers but wants to be cited in AI answers usually keeps the search and user-triggered crawlers allowed.

FAQ

How many websites block ClaudeBot?
As of 2026-11-16, 1,221 of the 8,313 popular websites whose robots.txt we could read (14.7%) block ClaudeBot, Anthropic's AI training crawler, on the home page; 228 more block it on some paths. Among the top 100 sites: 25.3%.
What is ClaudeBot?
ClaudeBot is Anthropic's AI training crawler: it collects pages to train AI models. It is the #4 most blocked of the 29 AI crawlers in the index.
How do I block ClaudeBot in robots.txt?
Add these lines to https://<your-site>/robots.txt: "User-agent: ClaudeBot" followed by "Disallow: /". To allow it, use "Allow: /" instead. Well-behaved crawlers read robots.txt before fetching pages; firewall rules are needed to stop crawlers that ignore it.
How do I check whether my site blocks ClaudeBot?
Use the free AI crawler checker at tools.yukai.uk/free/ai-crawler-checker: it shows the status of ClaudeBot and 28 other AI crawlers with the exact robots.txt rule.

Other AI crawlers

Also from Anthropic: Claude-SearchBot (5.7%), Claude-User (5.9%).