TidyToolsAI Crawler Index · data as of · updated daily

AI crawler · Google · Training

Which websites block Google-Extended?

As of 2026-11-16, 1,098 of the 8,313 popular websites whose robots.txt we could read (13.2%) block Google-Extended, Google's AI training crawler, on the home page; 204 more block it on some paths. Among the top 100 sites: 20.5%.

13.2%
of 8,313 readable sites block Google-Extended
20.5%
of the top 100
#7
most blocked of 29 AI crawlers
7,011
sites allow it on the home page

Blocked by category

CategorySites readBlock Google-Extended
Social5133.3%
News & media39228.3%
AI1020%
Entertainment33216.6%
Reference4316.3%
E-commerce11613.8%
Other665113%
Finance1027.8%
Travel137.7%
Education2324.3%
Tech1622.5%
Government2032%
Telecom60%

Popular sites that block Google-Extended (1,098)

Showing the 200 most popular of 1,098. The full list is in sites.csv.

Sites that block Google-Extended on some paths (204)

Showing the 60 most popular of 204. The full list is in sites.csv.

Trend

DateShare blocking Google-Extended
2026-11-1613.2%
2026-11-1513.2%
2026-11-1413.2%
2026-11-1313.2%
2026-11-1213.2%
2026-11-1113.2%
2026-11-1013.2%
2026-11-0913.2%
2026-11-0813.3%
2026-11-0713.4%
2026-11-0613.3%
2026-11-0513.3%
2026-11-0413.3%
2026-11-0313.3%
2026-11-0213.2%
2026-11-0113.2%
2026-10-3113.3%
2026-10-3013.2%
2026-10-2913.2%
2026-10-2812.7%
2026-10-2711.8%
2026-10-2611.2%
2026-10-2511.2%
2026-10-2411.1%
2026-10-2311%
2026-10-2211%
2026-10-2111.3%
2026-10-2010.5%
2026-10-1910.4%
2026-10-1810.2%

Latest changes

Block or allow Google-Extended

User-agent: Google-Extended
Disallow: /

Replace Disallow: / with Allow: / to let it in. A site that blocks training crawlers but wants to be cited in AI answers usually keeps the search and user-triggered crawlers allowed.

FAQ

How many websites block Google-Extended?
As of 2026-11-16, 1,098 of the 8,313 popular websites whose robots.txt we could read (13.2%) block Google-Extended, Google's AI training crawler, on the home page; 204 more block it on some paths. Among the top 100 sites: 20.5%.
What is Google-Extended?
Google-Extended is Google's AI training crawler: it collects pages to train AI models. It is the #7 most blocked of the 29 AI crawlers in the index.
How do I block Google-Extended in robots.txt?
Add these lines to https://<your-site>/robots.txt: "User-agent: Google-Extended" followed by "Disallow: /". To allow it, use "Allow: /" instead. Well-behaved crawlers read robots.txt before fetching pages; firewall rules are needed to stop crawlers that ignore it.
How do I check whether my site blocks Google-Extended?
Use the free AI crawler checker at tools.yukai.uk/free/ai-crawler-checker: it shows the status of Google-Extended and 28 other AI crawlers with the exact robots.txt rule.

Other AI crawlers

Also from Google: Google-CloudVertexBot (8.8%), Google-GeminiNotebook (3.3%), Google-Agent (3.6%).