TidyToolsAI Crawler Index · data as of · updated daily

AI Crawler Index · country

Do websites in Italy (.it) block AI crawlers?

As of 2026-10-03, 27.3% of the 55 websites in Italy (.it) whose robots.txt we could read block GPTBot, 25.5% block ClaudeBot and 21.8% block PerplexityBot (all 8,357 sites: 12%, 11%, 8.3%). 4 of them (7.3%) serve an llms.txt (all sites: 15.2%).

27.3%
block GPTBot (all sites 12%)
25.5%
block ClaudeBot (all sites 11%)
7.3%
serve an llms.txt (all sites 15.2%)
55
of 81 sites with a readable robots.txt

Block rate by crawler: .it (Italy) vs all sites

CrawlerOperator.it (Italy)All sitesDifference
CCBotCommon Crawl32.7%12.8%+19.9 pp
BytespiderByteDance30.9%12.3%+18.6 pp
GPTBotOpenAI27.3%12%+15.3 pp
ClaudeBotAnthropic25.5%11%+14.5 pp
Google-ExtendedGoogle23.6%9.5%+14.1 pp
AmazonbotAmazon21.8%9.2%+12.6 pp
cohere-aiCohere21.8%9.1%+12.7 pp
PerplexityBotPerplexity21.8%8.3%+13.5 pp
Applebot-ExtendedApple16.4%9.6%+6.8 pp
ChatGPT-UserOpenAI14.5%7.5%+7 pp
DuckAssistBotDuckDuckGo14.5%5.8%+8.7 pp
Meta-ExternalAgentMeta14.5%9.5%+5 pp
ApplebotApple12.7%3.1%+9.6 pp
Claude-SearchBotAnthropic7.3%5.4%+1.9 pp
Claude-UserAnthropic7.3%5.5%+1.8 pp
OAI-SearchBotOpenAI7.3%5.7%+1.6 pp
Google-CloudVertexBotGoogle5.5%5.2%+0.3 pp
meta-externalfetcherMeta5.5%6%-0.5 pp
Amzn-SearchBotAmazon0%4.1%-4.1 pp
Amzn-UserAmazon0%3.9%-3.9 pp
Google-AgentGoogle0%3.2%-3.2 pp
Google-GeminiNotebookGoogle0%3%-3 pp
meta-externaladsMeta0%2.8%-2.8 pp
meta-webindexerMeta0%4.6%-4.6 pp
MistralAI-IndexMistral0%3.3%-3.3 pp
MistralAI-TrainingMistral0%3.3%-3.3 pp
MistralAI-UserMistral0%5.4%-5.4 pp
OAI-AdsBotOpenAI0%3%-3 pp
Perplexity-UserPerplexity0%5.7%-5.7 pp

Who blocks what, by name: GPTBot, ClaudeBot, PerplexityBot, Google-Extended, CCBot, Bytespider, Applebot-Extended, OAI-SearchBot.

Policy mix

PolicySitesShare
Open3340.7%
Unreadable2632.1%
Blocks some AI search1518.5%
Blocks training only67.4%
No robots.txt11.2%

Most restrictive .it (Italy) sites

  1. ilmessaggero.it 14 of 29 blocked
  2. leggo.it 14 of 29 blocked
  3. ilmattino.it 14 of 29 blocked
  4. ilgazzettino.it 14 of 29 blocked
  5. ansa.it 12 of 29 blocked
  6. corriere.it 11 of 29 blocked
  7. repubblica.it 10 of 29 blocked
  8. giallozafferano.it 10 of 29 blocked
  9. my-personaltrainer.it 10 of 29 blocked
  10. rainews.it 9 of 29 blocked
  11. raiplay.it 9 of 29 blocked
  12. gazzetta.it 8 of 29 blocked
  13. ilmeteo.it 8 of 29 blocked
  14. diretta.it 6 of 29 blocked
  15. lastampa.it 6 of 29 blocked

Sites with an llms.txt (4)

All 81 .it (Italy) sites

✕ blocked · ◐ partially blocked (some paths) · ✓ allowed · ? robots.txt unreadable. "#" is the position in our popularity-ordered sample.

#SitePolicyGPTBotOAI-SearchChatGPT-UserClaudeBotClaude-SearchPerplexityGoogle-Ext.CCBotllms.txt
430repubblica.itBlocks some AI searchblockedallowedallowedallowedallowedblockedblockedblockedyes
433corriere.itBlocks some AI searchblockedallowedblockedblockedallowedallowedblockedblocked–
574gazzetta.itBlocks some AI searchblockedallowedallowedblockedallowedallowedblockedblocked–
621ilfattoquotidiano.itOpenallowedallowedallowedallowedallowedallowedallowedallowed–
640subito.itUnreadableunknownunknownunknownunknownunknownunknownunknownunknown–
673ilmeteo.itBlocks training onlyblockedpartialpartialblockedpartialpartialpartialblockedyes
682inps.itOpenallowedallowedallowedallowedallowedallowedallowedallowed–
729immobiliare.itOpenallowedallowedallowedallowedallowedallowedallowedallowed–
968diretta.itBlocks some AI searchallowedallowedallowedallowedallowedallowedallowedblocked–
1078redd.itOpenallowedallowedallowedallowedallowedallowedallowedallowed–
1190libero.itOpenallowedallowedallowedallowedallowedallowedallowedallowed–
1665kahoot.itOpenallowedallowedallowedallowedallowedallowedallowedallowed–
1800poste.itOpenallowedallowedallowedallowedallowedallowedallowedallowed–
1889ansa.itBlocks some AI searchblockedallowedblockedblockedallowedblockedblockedblocked–
2227aruba.itOpenallowedallowedallowedallowedallowedallowedallowedallowed–
2409nirvam.itOpenallowedallowedallowedallowedallowedallowedallowedallowed–
2449pin.itUnreadableunknownunknownunknownunknownunknownunknownunknownunknown–
2551lastampa.itBlocks some AI searchallowedallowedallowedallowedallowedblockedallowedallowed–
2559justpaste.itOpenallowedallowedallowedallowedallowedallowedallowedallowed–
2933mediaset.itOpenallowedallowedallowedallowedallowedallowedallowedallowed–
3074rainews.itBlocks some AI searchblockedallowedblockedblockedallowedblockedblockedblocked–
3315agenziaentrate.gov.itUnreadableunknownunknownunknownunknownunknownunknownunknownunknown–
3324unibo.itOpenallowedallowedallowedallowedallowedallowedallowedallowed–
3402ilmessaggero.itBlocks some AI searchblockedblockedblockedblockedblockedblockedblockedblocked–
3406garanteprivacy.itOpenallowedallowedallowedallowedallowedallowedallowedallowed–
3634raiplay.itBlocks some AI searchblockedallowedblockedblockedallowedblockedblockedblocked–
3679interno.gov.itUnreadableunknownunknownunknownunknownunknownunknownunknownunknown–
3710fanpage.itUnreadableunknownunknownunknownunknownunknownunknownunknownunknown–
3719ilpost.itBlocks some AI searchblockedallowedallowedallowedallowedblockedallowedallowed–
3741ilgiornale.itOpenallowedallowedallowedallowedallowedallowedallowedallowed–
4038esteri.itOpenallowedallowedallowedallowedallowedallowedallowedallowed–
4124trovaprezzi.itOpenallowedallowedallowedallowedallowedallowedallowedallowedyes
4235poliziadistato.itOpenallowedallowedallowedallowedallowedallowedallowedallowed–
4420virgilio.itOpenallowedallowedallowedallowedallowedallowedallowedallowed–
4424ilrestodelcarlino.itBlocks training onlyallowedallowedallowedblockedallowedallowedallowedblocked–
4480giallozafferano.itBlocks some AI searchblockedallowedallowedblockedallowedblockedblockedblocked–
4491lanazione.itBlocks training onlyallowedallowedallowedblockedallowedallowedallowedblocked–
4558borsaitaliana.itOpenallowedallowedallowedallowedallowedallowedallowedallowed–
4663sisal.itUnreadableunknownunknownunknownunknownunknownunknownunknownunknown–
4770snai.itUnreadableunknownunknownunknownunknownunknownunknownunknownunknown–
4781scoop.itNo robots.txtallowedallowedallowedallowedallowedallowedallowedallowed–
4893leggo.itBlocks some AI searchblockedblockedblockedblockedblockedblockedblockedblocked–
4901my-personaltrainer.itBlocks some AI searchblockedallowedallowedblockedallowedblockedblockedblocked–
4922tiscali.itOpenallowedallowedallowedallowedallowedallowedallowedallowed–
5030lefrecce.itOpenallowedallowedallowedallowedallowedallowedallowedallowed–
5042goldbet.itUnreadableunknownunknownunknownunknownunknownunknownunknownunknown–
5211lottomatica.itUnreadableunknownunknownunknownunknownunknownunknownunknownunknown–
5290ilmattino.itBlocks some AI searchblockedblockedblockedblockedblockedblockedblockedblocked–
5500ildolomiti.itBlocks training onlyblockedallowedallowedallowedallowedallowedblockedblocked–
5515betflag.itUnreadableunknownunknownunknownunknownunknownunknownunknownunknown–
5571iliad.itOpenallowedallowedallowedallowedallowedallowedallowedallowed–
5602planetwin365.itUnreadableunknownunknownunknownunknownunknownunknownunknownunknown–
5649fcinter1908.itOpenallowedallowedallowedpartialallowedallowedallowedallowed–
5835fantacalcio.itOpenallowedallowedallowedallowedallowedallowedallowedallowed–
6021tecnomat.itUnreadableunknownunknownunknownunknownunknownunknownunknownunknown–
6106mondosanita.itBlocks training onlyallowedallowedallowedallowedallowedallowedallowedblocked–
6214ilgazzettino.itBlocks some AI searchblockedblockedblockedblockedblockedblockedblockedblocked–
6420unimi.itUnreadableunknownunknownunknownunknownunknownunknownunknownunknown–
6633rai.itOpenallowedallowedallowedallowedallowedallowedallowedallowed–
6725unipd.itOpenallowedallowedallowedallowedallowedallowedallowedallowed–
6903milannews.itOpenallowedallowedallowedallowedallowedallowedallowedallowed–
6907infn.itUnreadableunknownunknownunknownunknownunknownunknownunknownunknown–
7026polimi.itUnreadableunknownunknownunknownunknownunknownunknownunknownunknown–
7108giornalone.itOpenallowedallowedallowedallowedallowedallowedallowedallowed–
7545uniroma1.itUnreadableunknownunknownunknownunknownunknownunknownunknownunknown–
7562paginegialle.itUnreadableunknownunknownunknownunknownunknownunknownunknownunknown–
8005unifi.itUnreadableunknownunknownunknownunknownunknownunknownunknownunknown–
8381follow.itOpenallowedallowedallowedallowedallowedallowedallowedallowed–
8451unipi.itOpenallowedallowedallowedallowedallowedallowedallowedallowed–
8579cineca.itUnreadableunknownunknownunknownunknownunknownunknownunknownunknown–
8855unito.itUnreadableunknownunknownunknownunknownunknownunknownunknownunknown–
9181nanosystems.itOpenallowedallowedallowedallowedallowedallowedallowedallowed–
9545casa.itOpenallowedallowedallowedallowedallowedallowedallowedallowed–
9583gazzettaufficiale.itUnreadableunknownunknownunknownunknownunknownunknownunknownunknown–
9590lafeltrinelli.itBlocks training onlyallowedallowedallowedallowedallowedallowedallowedallowed–
9630corrieredellosport.itOpenallowedallowedallowedallowedallowedallowedallowedallowedyes
9661glauco.itUnreadableunknownunknownunknownunknownunknownunknownunknownunknown–
9822salute.gov.itUnreadableunknownunknownunknownunknownunknownunknownunknownunknown–
9840unina.itUnreadableunknownunknownunknownunknownunknownunknownunknownunknown–
9856mim.gov.itUnreadableunknownunknownunknownunknownunknownunknownunknownunknown–
9917mef.gov.itUnreadableunknownunknownunknownunknownunknownunknownunknownunknown–

Categories on .it (Italy)

FAQ

Do websites in Italy (.it) block AI crawlers?
As of 2026-10-03, 27.3% of the 55 websites in Italy (.it) whose robots.txt we could read block GPTBot, 25.5% block ClaudeBot and 21.8% block PerplexityBot (all 8,357 sites: 12%, 11%, 8.3%). 4 of them (7.3%) serve an llms.txt (all sites: 15.2%).
Which .it (Italy) websites block the most AI crawlers?
ilmessaggero.it (14 of 29), leggo.it (14 of 29), ilmattino.it (14 of 29), ilgazzettino.it (14 of 29), ansa.it (12 of 29). The full per-site status is on each site page.
How is this measured?
We fetch each site's robots.txt every day from one residential IP in Taiwan with an honest user agent, and apply the rules to 29 AI crawlers for the home page. 81 .it (Italy) sites are in the sample, 55 had a readable robots.txt (26 did not answer or returned an error page).

Other countries and suffixes

Compare all suffixes