TidyToolsAI Crawler Index · data as of · updated daily

AI crawler · Anthropic · AI search

Which websites block Claude-SearchBot?

As of 2026-10-01, 69 of the 853 popular websites whose robots.txt we could read (8.1%) block Claude-SearchBot, Anthropic's AI search crawler, on the home page; 38 more block it on some paths. Among the top 100 sites: 12%.

8.1%
of 853 readable sites block Claude-SearchBot
12%
of the top 100
#16
most blocked of 29 AI crawlers
746
sites allow it on the home page

Blocked by category

CategorySites readBlock Claude-SearchBot
News & media12628.6%
Social4818.8%
Travel137.7%
Reference437%
Other2605.8%
E-commerce585.2%
Tech1541.3%
AI90%
Education200%
Entertainment680%
Finance160%
Government330%
Telecom50%

Popular sites that block Claude-SearchBot (69)

  1. facebook.com Social
  2. instagram.com Social
  3. twitter.com Social
  4. amazon.com E-commerce
  5. x.com Social
  6. pinterest.com Social
  7. tiktok.com Social
  8. reddit.com Social
  9. nytimes.com News & media
  10. theguardian.com News & media
  11. meraki.com Tech
  12. imdb.com Reference
  13. launchpad.net Other
  14. reuters.com News & media
  15. weibo.com Social
  16. wsj.com News & media
  17. unsplash.com Tech
  18. quora.com Reference
  19. wired.com News & media
  20. uol.com.br News & media
  21. usatoday.com News & media
  22. threads.com Social
  23. telegraph.co.uk News & media
  24. elpais.com News & media
  25. dailymail.com News & media
  26. lemonde.fr News & media
  27. fb.com Other
  28. cnet.com News & media
  29. apnews.com News & media
  30. people.com News & media
  31. theverge.com News & media
  32. nbcnews.com News & media
  33. investopedia.com News & media
  34. theatlantic.com News & media
  35. as.com News & media
  36. bfmtv.com News & media
  37. huffpost.com News & media
  38. aftonbladet.se News & media
  39. gsmarena.com News & media
  40. nypost.com News & media
  41. economist.com News & media
  42. theconversation.com News & media
  43. vg.no News & media
  44. spiegel.de News & media
  45. amap.com Travel
  46. emag.ro E-commerce
  47. themoviedb.org Reference
  48. actu.fr News & media
  49. huffingtonpost.com News & media
  50. sapo.pt News & media
  51. iltalehti.fi News & media
  52. ekstrabladet.dk News & media
  53. tjk.org Other
  54. aka.ms Other
  55. sportfilm800.com News & media
  56. rocketmoney.dev Other
  57. qcloud.com Other
  58. newyorker.com Other
  59. blackhub.team Other
  60. arstechnica.com News & media
  61. zdnet.com News & media
  62. adrta.com Other
  63. marketwatch.com E-commerce
  64. scientificamerican.com Other
  65. wikihow.com Other
  66. sogou.com Other
  67. ign.com Other
  68. salesforceliveagent.com Other
  69. sedoparking.com Other

Sites that block Claude-SearchBot on some paths (38)

  1. linkedin.com Social
  2. netflix.com Entertainment
  3. yahoo.com Tech
  4. snapchat.com Social
  5. flickr.com Other
  6. ebay.com E-commerce
  7. android.com Tech
  8. ubuntu.com Tech
  9. hostinger.com Tech
  10. trustpilot.com Other
  11. bloomberg.com News & media
  12. indeed.com Other
  13. temu.com E-commerce
  14. paloaltonetworks.com Tech
  15. sohu.com Tech
  16. capcut.com Other
  17. bitrix24.ru Other
  18. trendmicro.com Tech
  19. klaviyo.com Tech
  20. ted.com Entertainment
  21. disneyplus.com Entertainment
  22. academia.edu Reference
  23. figma.com Tech
  24. expedia.com Travel
  25. jimdo.com Tech
  26. skroutz.gr E-commerce
  27. leboncoin.fr E-commerce
  28. bbb.org Other
  29. dagbladet.no News & media
  30. ilmeteo.it News & media
  31. deezer.com Other
  32. meteociel.fr News & media
  33. shutterstock.com Other
  34. merkur.de Other
  35. hulu.com Entertainment
  36. oreilly.com Other
  37. fastcompany.com Other
  38. lnkd.in Other

Trend

The daily series for Claude-SearchBot starts on 2026-10-01; a trend appears here as days accumulate.

Latest changes

Block or allow Claude-SearchBot

User-agent: Claude-SearchBot
Disallow: /

Replace Disallow: / with Allow: / to let it in. A site that blocks training crawlers but wants to be cited in AI answers usually keeps the search and user-triggered crawlers allowed.

FAQ

How many websites block Claude-SearchBot?
As of 2026-10-01, 69 of the 853 popular websites whose robots.txt we could read (8.1%) block Claude-SearchBot, Anthropic's AI search crawler, on the home page; 38 more block it on some paths. Among the top 100 sites: 12%.
What is Claude-SearchBot?
Claude-SearchBot is Anthropic's AI search crawler: it indexes pages so an AI search product can find and cite them. It is the #16 most blocked of the 29 AI crawlers in the index.
How do I block Claude-SearchBot in robots.txt?
Add these lines to https://<your-site>/robots.txt: "User-agent: Claude-SearchBot" followed by "Disallow: /". To allow it, use "Allow: /" instead. Well-behaved crawlers read robots.txt before fetching pages; firewall rules are needed to stop crawlers that ignore it.
How do I check whether my site blocks Claude-SearchBot?
Use the free AI crawler checker at tools.yukai.uk/free/ai-crawler-checker: it shows the status of Claude-SearchBot and 28 other AI crawlers with the exact robots.txt rule.

Other AI crawlers

Also from Anthropic: ClaudeBot (16.3%), Claude-User (8.3%).