AI crawler · Google · AI search
Which websites block Google-CloudVertexBot?
As of 2026-11-16, 733 of the 8,313 popular websites whose robots.txt we could read (8.8%) block Google-CloudVertexBot, Google's AI search crawler, on the home page; 27 more block it on some paths. Among the top 100 sites: 16.9%.
8.8%
of 8,313 readable sites block Google-CloudVertexBot
16.9%
of the top 100
#10
most blocked of 29 AI crawlers
7,553
sites allow it on the home page
Blocked by category
| Category | Sites read | Block Google-CloudVertexBot |
|---|---|---|
| Social | 51 | 23.5% |
| Travel | 13 | 15.4% |
| Entertainment | 332 | 13.9% |
| News & media | 392 | 12.8% |
| Reference | 43 | 9.3% |
| Other | 6651 | 8.9% |
| E-commerce | 116 | 7.8% |
| Finance | 102 | 4.9% |
| Education | 232 | 3.9% |
| Government | 203 | 2% |
| AI | 10 | 0% |
| Tech | 162 | 0% |
| Telecom | 6 | 0% |
Popular sites that block Google-CloudVertexBot (733)
- facebook.com Social
- instagram.com Social
- twitter.com Social
- linkedin.com Social
- amazon.com E-commerce
- netflix.com Entertainment
- x.com Social
- pinterest.com Social
- reddit.com Social
- amazonvideo.com Entertainment
- nytimes.com News & media
- flickr.com Other
- theguardian.com News & media
- bbc.com News & media
- imdb.com Reference
- launchpad.net Other
- reuters.com News & media
- weibo.com Social
- weather.com News & media
- trustpilot.com Other
- bloomberg.com News & media
- wired.com News & media
- tripadvisor.com Travel
- usatoday.com News & media
- threads.com Social
- telegraph.co.uk News & media
- primevideo.com Entertainment
- dailymail.com News & media
- kick.com Entertainment
- lemonde.fr News & media
- fb.com Other
- patreon.com Social
- disneyplus.com Entertainment
- people.com News & media
- lefigaro.fr News & media
- academia.edu Reference
- theverge.com News & media
- corriere.it News & media
- investopedia.com News & media
- elmundo.es News & media
- ouest-france.fr News & media
- bfmtv.com News & media
- marca.com News & media
- theconversation.com News & media
- gazzetta.it News & media
- amap.com Travel
- jusbrasil.com.br Other
- litnet.com Other
- themoviedb.org Reference
- mydramalist.com Reference
- dagbladet.no News & media
- iltalehti.fi News & media
- cardmarket.com E-commerce
- dogdrip.net Social
- tjk.org Other
- aka.ms Other
- myreadingmanga.info Entertainment
- theqoo.net Other
- twkan.com Other
- sportfilm800.com News & media
- qcloud.com Other
- newyorker.com Other
- arstechnica.com News & media
- rocketmoney.dev Other
- adrta.com Other
- salthasher.com Other
- blackhub.team Other
- nontondrama.my Other
- lnkd.in Other
- vidoyu.space Other
- jhu.edu Education
- sedoparking.com Other
- sogou.com Other
- bollyflix.free Other
- salesforceliveagent.com Other
- adsco.re Other
- videeyss.shop E-commerce
- cukrarnauandela.cz Other
- moviezwap.taxi Entertainment
- filmyzilla63.com Entertainment
- filmyzilla56.com Entertainment
- filmyzilla62.com Entertainment
- thesun.co.uk Other
- sap.com Other
- openssl.org Other
- constellix.com Other
- kasserver.com Other
- vox.com Other
- instacart.com Other
- congress.gov Government
- co.com Other
- guardian.co.uk Other
- tappx.com Other
- redditmedia.com Other
- zeit.de Other
- expressapisv2.net News & media
- smh.com.au Other
- technologyreview.com Other
- urldefense.com Other
- thehill.com Other
- tmdb.org Other
- gutenberg.org Other
- ko-fi.com Other
- theglobeandmail.com Other
- gamespot.com Entertainment
- nextdoor.com Social
- huntress.io Other
- system-monitor.com Other
- newscientist.com News & media
- franceinfo.fr Other
- salesforce-scrt.com Other
- mastodon.social Other
- bizjournals.com News & media
- realtor.com Other
- glassdoor.com Other
- wa.gov Government
- jwpltx.com Other
- imageshack.us Other
- asahi.com Other
- nrk.no Other
- thetimes.co.uk News & media
- intelbras.com.br Other
- yumpu.com Other
- hellotalk8.com Other
- theregister.com Other
- sas.com Other
- vogue.com Other
- metro.co.uk Other
- unep.org Other
- allrecipes.com Other
- wargaming.net Other
- nzherald.co.nz News & media
- cuny.edu Education
- nymag.com Other
- vietnam.vn Other
- lenovomm.com Other
- ansa.it Other
- nos.nl Other
- smugmug.com Other
- rs6.net Other
- thefreedictionary.com Other
- onenote.com Other
- vanityfair.com Other
- entrust.net Other
- wiki.gg Other
- oregonstate.edu Education
- sportradarserving.com News & media
- backblazeb2.com Other
- 255md.com Other
- codepen.io Other
- findagrave.com Other
- vaticannews.va News & media
- wordreference.com Other
- gmarket.co.kr E-commerce
- thestar.com Other
- seekingalpha.com Other
- seattletimes.com News & media
- derstandard.at Other
- artfight.net Other
- radiofrance.fr Other
- cnki.net Other
- wattloom.com Other
- blender.org Other
- lectormangass.net Entertainment
- filmyzilla52.com Entertainment
- rexify.com.ng Other
- computerworld.com Other
- verywellhealth.com Other
- myhostadmin.net Other
- helsinki.fi Other
- paycomonline.net Finance
- ew.com Other
- verywellmind.com Other
- southernliving.com Other
- feednews.com News & media
- huya.com Other
- doh.pub Other
- nine.com.au Other
- ezoic.net Other
- ap.org Other
- eatingwell.com Other
- boxofficemojo.com Other
- lycos.com Other
- telegraaf.nl Other
- channel4.com Other
- rm358.com Other
- dayforcehcm.com Other
- arte.tv Entertainment
- pscp.tv Entertainment
- codeforces.com Other
- inaturalist.org Other
- michelin.com Other
- gq.com Other
- theage.com.au Other
- expressen.se News & media
- travelandleisure.com Other
- rutracker.org Other
- csmonitor.com Other
- marthastewart.com Other
- upi.com Other
Showing the 200 most popular of 733. The full list is in sites.csv.
Sites that block Google-CloudVertexBot on some paths (27)
- tradingview.com News & media
- wsj.com News & media
- statista.com News & media
- sohu.com Tech
- change.org Social
- jimdo.com Tech
- hostinger.com Tech
- merkur.de Other
- marketwatch.com E-commerce
- newsbreak.com News & media
- alltrails.com Other
- barrons.com Other
- 20min.ch Other
- ntv.co.jp Other
- fr.de Other
- manageengine.com Other
- njuskalo.hr Other
- ruhr24.de Other
- doctoreto.com Other
- bebee.com Other
- wral.com Other
- investors.com Finance
- unfpa.org Other
- justanswer.com Other
- auspost.com.au News & media
- hna.de Other
- sonarsource.com Other
Trend
The daily series for Google-CloudVertexBot starts on 2026-11-16; a trend appears here as days accumulate.
Latest changes
- gutenberg.org: allowed → blocked
- athome.co.jp: allowed → blocked
- yclients.com: allowed → blocked
- dreamwidth.org: allowed → blocked
- passportindia.gov.in: blocked → allowed
Block or allow Google-CloudVertexBot
User-agent: Google-CloudVertexBot Disallow: /
Replace Disallow: / with Allow: / to let it in. A site that blocks training crawlers but wants to be cited in AI answers usually keeps the search and user-triggered crawlers allowed.
FAQ
- How many websites block Google-CloudVertexBot?
- As of 2026-11-16, 733 of the 8,313 popular websites whose robots.txt we could read (8.8%) block Google-CloudVertexBot, Google's AI search crawler, on the home page; 27 more block it on some paths. Among the top 100 sites: 16.9%.
- What is Google-CloudVertexBot?
- Google-CloudVertexBot is Google's AI search crawler: it indexes pages so an AI search product can find and cite them. It is the #10 most blocked of the 29 AI crawlers in the index.
- How do I block Google-CloudVertexBot in robots.txt?
- Add these lines to https://<your-site>/robots.txt: "User-agent: Google-CloudVertexBot" followed by "Disallow: /". To allow it, use "Allow: /" instead. Well-behaved crawlers read robots.txt before fetching pages; firewall rules are needed to stop crawlers that ignore it.
- How do I check whether my site blocks Google-CloudVertexBot?
- Use the free AI crawler checker at tools.yukai.uk/free/ai-crawler-checker: it shows the status of Google-CloudVertexBot and 28 other AI crawlers with the exact robots.txt rule.
Other AI crawlers
Also from Google: Google-Extended (13.2%), Google-GeminiNotebook (3.3%), Google-Agent (3.6%).
- CCBot 16.4%
- Bytespider 16.2%
- GPTBot 15.4%
- ClaudeBot 14.7%
- Meta-ExternalAgent 13.6%
- Applebot-Extended 13.4%
- Google-Extended 13.2%
- Amazonbot 13.2%
- cohere-ai 12.7%
- PerplexityBot 8.2%
- ChatGPT-User 7.4%
- MistralAI-Training 7.1%
- meta-externalfetcher 6.5%
- Perplexity-User 6.1%
- Claude-User 5.9%
- DuckAssistBot 5.9%
- OAI-SearchBot 5.8%
- Claude-SearchBot 5.7%
- MistralAI-User 5.7%
- meta-webindexer 5.1%
- Amzn-SearchBot 4.4%
- Amzn-User 4.4%
- Google-Agent 3.6%
- MistralAI-Index 3.6%
- Google-GeminiNotebook 3.3%
- Applebot 3.2%
- OAI-AdsBot 3%
- meta-externalads 2.9%