Claude-User is Anthropic's user-triggered AI fetcher: it fetches a page when a user asks the assistant about it. It is the #15 most blocked of the 29 AI crawlers in the index.

How do I block Claude-User in robots.txt?

Add these lines to https:// /robots.txt: "User-agent: Claude-User" followed by "Disallow: /". To allow it, use "Allow: /" instead. Well-behaved crawlers read robots.txt before fetching pages; firewall rules are needed to stop crawlers that ignore it.

How do I check whether my site blocks Claude-User?

Use the free AI crawler checker at tools.yukai.uk/free/ai-crawler-checker: it shows the status of Claude-User and 28 other AI crawlers with the exact robots.txt rule.

AI crawler · Anthropic · User-triggered fetch

Which websites block Claude-User?

As of 2026-10-01, 71 of the 853 popular websites whose robots.txt we could read (8.3%) block Claude-User, Anthropic's user-triggered AI fetcher, on the home page; 31 more block it on some paths. Among the top 100 sites: 14.5%.

8.3%

of 853 readable sites block Claude-User

14.5%

of the top 100

#15

most blocked of 29 AI crawlers

751

sites allow it on the home page

Blocked by category

Category	Sites read	Block Claude-User
News & media	126	27%
Social	48	20.8%
Travel	13	7.7%
Reference	43	7%
Other	260	6.5%
E-commerce	58	5.2%
Entertainment	68	2.9%
Tech	154	0.6%
AI	9	0%
Education	20	0%
Finance	16	0%
Government	33	0%
Telecom	5	0%

Trend

The daily series for Claude-User starts on 2026-10-01; a trend appears here as days accumulate.

Latest changes

dailymotion.com: blocked → allowed
actu.fr: allowed → blocked
scientificamerican.com: allowed → blocked

Block or allow Claude-User

User-agent: Claude-User
Disallow: /

Replace Disallow: / with Allow: / to let it in. A site that blocks training crawlers but wants to be cited in AI answers usually keeps the search and user-triggered crawlers allowed.

FAQ

How many websites block Claude-User?: As of 2026-10-01, 71 of the 853 popular websites whose robots.txt we could read (8.3%) block Claude-User, Anthropic's user-triggered AI fetcher, on the home page; 31 more block it on some paths. Among the top 100 sites: 14.5%.
What is Claude-User?: Claude-User is Anthropic's user-triggered AI fetcher: it fetches a page when a user asks the assistant about it. It is the #15 most blocked of the 29 AI crawlers in the index.
How do I block Claude-User in robots.txt?: Add these lines to https://<your-site>/robots.txt: "User-agent: Claude-User" followed by "Disallow: /". To allow it, use "Allow: /" instead. Well-behaved crawlers read robots.txt before fetching pages; firewall rules are needed to stop crawlers that ignore it.
How do I check whether my site blocks Claude-User?: Use the free AI crawler checker at tools.yukai.uk/free/ai-crawler-checker: it shows the status of Claude-User and 28 other AI crawlers with the exact robots.txt rule.

Other AI crawlers

Also from Anthropic: ClaudeBot (16.3%), Claude-SearchBot (8.1%).