TidyToolsAI Crawler Index · data as of · updated daily

AI crawler · Google · User-triggered fetch

Which websites block Google-Agent?

As of 2026-10-01, 38 of the 853 popular websites whose robots.txt we could read (4.5%) block Google-Agent, Google's user-triggered AI fetcher, on the home page; 6 more block it on some paths. Among the top 100 sites: 10.8%.

4.5%
of 853 readable sites block Google-Agent
10.8%
of the top 100
#25
most blocked of 29 AI crawlers
809
sites allow it on the home page

Blocked by category

CategorySites readBlock Google-Agent
Social4818.8%
Travel137.7%
Other2605.8%
News & media1265.6%
Reference434.7%
E-commerce583.4%
Entertainment681.5%
Tech1540.6%
AI90%
Education200%
Finance160%
Government330%
Telecom50%

Popular sites that block Google-Agent (38)

  1. facebook.com Social
  2. instagram.com Social
  3. twitter.com Social
  4. linkedin.com Social
  5. netflix.com Entertainment
  6. x.com Social
  7. pinterest.com Social
  8. reddit.com Social
  9. flickr.com Other
  10. meraki.com Tech
  11. imdb.com Reference
  12. reuters.com News & media
  13. weibo.com Social
  14. trustpilot.com Other
  15. wsj.com News & media
  16. cnbc.com News & media
  17. usatoday.com News & media
  18. threads.com Social
  19. lemonde.fr News & media
  20. fb.com Other
  21. academia.edu Reference
  22. theconversation.com News & media
  23. amap.com Travel
  24. emag.ro E-commerce
  25. tjk.org Other
  26. aka.ms Other
  27. sportfilm800.com News & media
  28. twkan.com Other
  29. rocketmoney.dev Other
  30. qcloud.com Other
  31. blackhub.team Other
  32. adrta.com Other
  33. marketwatch.com E-commerce
  34. merkur.de Other
  35. lnkd.in Other
  36. sogou.com Other
  37. salesforceliveagent.com Other
  38. sedoparking.com Other

Sites that block Google-Agent on some paths (6)

  1. indeed.com Other
  2. sohu.com Tech
  3. capcut.com Other
  4. klaviyo.com Tech
  5. disneyplus.com Entertainment
  6. hulu.com Entertainment

Trend

The daily series for Google-Agent starts on 2026-10-01; a trend appears here as days accumulate.

Latest changes

Block or allow Google-Agent

User-agent: Google-Agent
Disallow: /

Replace Disallow: / with Allow: / to let it in. A site that blocks training crawlers but wants to be cited in AI answers usually keeps the search and user-triggered crawlers allowed.

FAQ

How many websites block Google-Agent?
As of 2026-10-01, 38 of the 853 popular websites whose robots.txt we could read (4.5%) block Google-Agent, Google's user-triggered AI fetcher, on the home page; 6 more block it on some paths. Among the top 100 sites: 10.8%.
What is Google-Agent?
Google-Agent is Google's user-triggered AI fetcher: it fetches a page when a user asks the assistant about it. It is the #25 most blocked of the 29 AI crawlers in the index.
How do I block Google-Agent in robots.txt?
Add these lines to https://<your-site>/robots.txt: "User-agent: Google-Agent" followed by "Disallow: /". To allow it, use "Allow: /" instead. Well-behaved crawlers read robots.txt before fetching pages; firewall rules are needed to stop crawlers that ignore it.
How do I check whether my site blocks Google-Agent?
Use the free AI crawler checker at tools.yukai.uk/free/ai-crawler-checker: it shows the status of Google-Agent and 28 other AI crawlers with the exact robots.txt rule.

Other AI crawlers

Also from Google: Google-Extended (14%), Google-CloudVertexBot (7.2%), Google-GeminiNotebook (4.3%).