TidyToolsAI Crawler Index · data as of · updated daily

AI crawler · Anthropic · AI search

Which websites block Claude-SearchBot?

As of 2026-11-16, 471 of the 8,313 popular websites whose robots.txt we could read (5.7%) block Claude-SearchBot, Anthropic's AI search crawler, on the home page; 160 more block it on some paths. Among the top 100 sites: 13.3%.

5.7%
of 8,313 readable sites block Claude-SearchBot
13.3%
of the top 100
#19
most blocked of 29 AI crawlers
7,682
sites allow it on the home page

Blocked by category

CategorySites readBlock Claude-SearchBot
News & media39219.6%
Social5119.6%
Travel137.7%
Reference437%
Other66515.3%
E-commerce1164.3%
Finance1023.9%
Entertainment3322.7%
Government2032%
Education2320.9%
Tech1620.6%
AI100%
Telecom60%

Popular sites that block Claude-SearchBot (471)

Showing the 200 most popular of 471. The full list is in sites.csv.

Sites that block Claude-SearchBot on some paths (160)

Showing the 60 most popular of 160. The full list is in sites.csv.

Trend

DateShare blocking Claude-SearchBot
2026-11-165.7%
2026-11-155.6%
2026-11-145.6%
2026-11-135.6%
2026-11-125.6%
2026-11-115.6%
2026-11-105.6%
2026-11-095.6%
2026-11-085.6%
2026-11-075.7%
2026-11-065.7%
2026-11-055.6%
2026-11-045.7%
2026-11-035.7%
2026-11-025.6%
2026-11-015.6%
2026-10-315.6%
2026-10-305.6%
2026-10-295.6%
2026-10-285.6%
2026-10-275.6%
2026-10-265.6%
2026-10-255.6%
2026-10-245.5%
2026-10-235.5%
2026-10-225.7%
2026-10-216.1%
2026-10-205.5%
2026-10-195.5%
2026-10-185.5%

Latest changes

Block or allow Claude-SearchBot

User-agent: Claude-SearchBot
Disallow: /

Replace Disallow: / with Allow: / to let it in. A site that blocks training crawlers but wants to be cited in AI answers usually keeps the search and user-triggered crawlers allowed.

FAQ

How many websites block Claude-SearchBot?
As of 2026-11-16, 471 of the 8,313 popular websites whose robots.txt we could read (5.7%) block Claude-SearchBot, Anthropic's AI search crawler, on the home page; 160 more block it on some paths. Among the top 100 sites: 13.3%.
What is Claude-SearchBot?
Claude-SearchBot is Anthropic's AI search crawler: it indexes pages so an AI search product can find and cite them. It is the #19 most blocked of the 29 AI crawlers in the index.
How do I block Claude-SearchBot in robots.txt?
Add these lines to https://<your-site>/robots.txt: "User-agent: Claude-SearchBot" followed by "Disallow: /". To allow it, use "Allow: /" instead. Well-behaved crawlers read robots.txt before fetching pages; firewall rules are needed to stop crawlers that ignore it.
How do I check whether my site blocks Claude-SearchBot?
Use the free AI crawler checker at tools.yukai.uk/free/ai-crawler-checker: it shows the status of Claude-SearchBot and 28 other AI crawlers with the exact robots.txt rule.

Other AI crawlers

Also from Anthropic: ClaudeBot (14.7%), Claude-User (5.9%).