TidyToolsAI Crawler Index · data as of · updated daily

AI Crawler Index · domain suffix

Do .ai websites block AI crawlers?

As of 2026-10-02, 4.8% of the 62 .ai websites whose robots.txt we could read block GPTBot, 3.2% block ClaudeBot and 1.6% block PerplexityBot (all 8,366 sites: 12%, 11%, 8.3%). 26 of them (41.9%) serve an llms.txt (all sites: 15.2%).

4.8%
block GPTBot (all sites 12%)
3.2%
block ClaudeBot (all sites 11%)
41.9%
serve an llms.txt (all sites 15.2%)
62
of 64 sites with a readable robots.txt

Block rate by crawler: .ai vs all sites

CrawlerOperator.aiAll sitesDifference
GPTBotOpenAI4.8%12%-7.2 pp
AmazonbotAmazon3.2%9.2%-6 pp
Applebot-ExtendedApple3.2%9.6%-6.4 pp
BytespiderByteDance3.2%12.2%-9 pp
ChatGPT-UserOpenAI3.2%7.4%-4.2 pp
ClaudeBotAnthropic3.2%11%-7.8 pp
Google-ExtendedGoogle3.2%9.5%-6.3 pp
OAI-SearchBotOpenAI3.2%5.7%-2.5 pp
Amzn-SearchBotAmazon1.6%4.1%-2.5 pp
Amzn-UserAmazon1.6%3.9%-2.3 pp
ApplebotApple1.6%3.1%-1.5 pp
CCBotCommon Crawl1.6%12.7%-11.1 pp
Claude-SearchBotAnthropic1.6%5.4%-3.8 pp
Claude-UserAnthropic1.6%5.5%-3.9 pp
cohere-aiCohere1.6%9%-7.4 pp
DuckAssistBotDuckDuckGo1.6%5.7%-4.1 pp
Google-AgentGoogle1.6%3.2%-1.6 pp
Google-CloudVertexBotGoogle1.6%5.2%-3.6 pp
Google-GeminiNotebookGoogle1.6%3%-1.4 pp
meta-externaladsMeta1.6%2.8%-1.2 pp
Meta-ExternalAgentMeta1.6%9.5%-7.9 pp
meta-externalfetcherMeta1.6%6%-4.4 pp
meta-webindexerMeta1.6%4.6%-3 pp
MistralAI-IndexMistral1.6%3.3%-1.7 pp
MistralAI-TrainingMistral1.6%3.2%-1.6 pp
MistralAI-UserMistral1.6%5.3%-3.7 pp
OAI-AdsBotOpenAI1.6%3%-1.4 pp
Perplexity-UserPerplexity1.6%5.7%-4.1 pp
PerplexityBotPerplexity1.6%8.3%-6.7 pp

Who blocks what, by name: GPTBot.

Policy mix

PolicySitesShare
Open4976.6%
No robots.txt914.1%
Unreadable23.1%
Blocks training only23.1%
Blocks some AI search11.6%
Blocks all AI search11.6%

Most restrictive .ai sites

  1. mgln.ai 29 of 29 blocked
  2. claude.ai 4 of 29 blocked
  3. tripo3d.ai 3 of 29 blocked
  4. scite.ai 2 of 29 blocked

Sites with an llms.txt (26)

All 64 .ai sites

✕ blocked · ◐ partially blocked (some paths) · ✓ allowed · ? robots.txt unreadable. "#" is the position in our popularity-ordered sample.

#SitePolicyGPTBotOAI-SearchChatGPT-UserClaudeBotClaude-SearchPerplexityGoogle-Ext.CCBotllms.txt
185claude.aiBlocks some AI searchblockedblockedblockedallowedallowedallowedblockedallowed–
300perplexity.aiOpenallowedallowedallowedallowedallowedallowedallowedallowed–
1203axon.aiOpenallowedallowedallowedallowedallowedallowedallowedallowed–
1490swishapps.aiNo robots.txtallowedallowedallowedallowedallowedallowedallowedallowed–
1613e-volution.aiNo robots.txtallowedallowedallowedallowedallowedallowedallowedallowed–
1702hybrid.aiOpenallowedallowedallowedallowedallowedallowedallowedallowed–
1878sardine.aiOpenallowedallowedallowedallowedallowedallowedallowedallowedyes
2027bale.aiOpenallowedallowedallowedallowedallowedallowedallowedallowed–
2151openrouter.aiOpenallowedallowedallowedallowedallowedallowedallowedallowedyes
2179polyspeak.aiNo robots.txtallowedallowedallowedallowedallowedallowedallowedallowed–
2259learnings.aiOpenallowedallowedallowedallowedallowedallowedallowedallowed–
2367vdo.aiOpenallowedallowedallowedallowedallowedallowedallowedallowed–
2520higgsfield.aiOpenpartialpartialpartialpartialallowedpartialpartialallowedyes
2709x.aiOpenallowedallowedallowedallowedallowedallowedallowedallowed–
2754seaart.aiOpenallowedallowedallowedallowedallowedallowedallowedallowedyes
3393crushon.aiOpenallowedallowedallowedallowedallowedallowedallowedallowed–
3549beacons.aiOpenallowedallowedallowedallowedallowedallowedallowedallowedyes
3621nrich.aiOpenallowedallowedallowedallowedallowedallowedallowedallowedyes
3865drift-pixel.aiOpenallowedallowedallowedallowedallowedallowedallowedallowedyes
3975opencode.aiOpenallowedallowedallowedallowedallowedallowedallowedallowed–
4015anota.aiUnreadableunknownunknownunknownunknownunknownunknownunknownunknown–
4083programmaticx.aiOpenallowedallowedallowedallowedallowedallowedallowedallowedyes
4153spicychat.aiOpenallowedallowedallowedallowedallowedallowedallowedallowed–
4283granola.aiOpenallowedallowedallowedallowedallowedallowedallowedallowedyes
4313mistral.aiOpenallowedallowedallowedallowedallowedallowedallowedallowedyes
4546z.aiOpenallowedallowedallowedallowedallowedallowedallowedallowed–
4719nexverse.aiOpenallowedallowedallowedallowedallowedallowedallowedallowed–
4748tripo3d.aiBlocks training onlyallowedallowedallowedallowedallowedallowedallowedallowedyes
4760qwen.aiNo robots.txtallowedallowedallowedallowedallowedallowedallowedallowed–
4876landingsite.aiOpenallowedallowedallowedallowedallowedallowedallowedallowed–
5050loudecho.aiNo robots.txtallowedallowedallowedallowedallowedallowedallowedallowed–
5287otter.aiOpenallowedallowedallowedallowedallowedallowedallowedallowed–
5979verisoul.aiOpenallowedallowedallowedallowedallowedallowedallowedallowedyes
6124turboscribe.aiOpenallowedallowedallowedallowedallowedallowedallowedallowed–
6143scite.aiBlocks training onlyblockedallowedallowedblockedallowedallowedallowedallowedyes
6178masteredu.aiNo robots.txtallowedallowedallowedallowedallowedallowedallowedallowed–
6258mgln.aiBlocks all AI searchblockedblockedblockedblockedblockedblockedblockedblocked–
6666krisp.aiOpenallowedallowedallowedallowedallowedallowedallowedallowedyes
6935ourdream.aiOpenallowedallowedallowedallowedallowedallowedallowedallowedyes
7158engenius.aiOpenallowedallowedallowedallowedallowedallowedallowedallowed–
7264securiti.aiOpenallowedallowedallowedallowedallowedallowedallowedallowedyes
7486linke.aiOpenallowedallowedallowedallowedallowedallowedallowedallowed–
7532bluexad.aiNo robots.txtallowedallowedallowedallowedallowedallowedallowedallowed–
7554candy.aiOpenallowedallowedallowedallowedallowedallowedallowedallowed–
7620gmgn.aiOpenpartialpartialpartialpartialpartialpartialpartialallowedyes
7679superops.aiOpenallowedallowedallowedallowedallowedallowedallowedallowedyes
7763jasper.aiOpenallowedallowedallowedallowedallowedallowedallowedallowedyes
7791shortpixel.aiNo robots.txtallowedallowedallowedallowedallowedallowedallowedallowed–
7969luciq.aiOpenallowedallowedallowedallowedallowedallowedallowedallowedyes
7991trae.aiOpenallowedallowedallowedallowedallowedallowedallowedallowed–
8150arena.aiOpenallowedallowedallowedallowedallowedallowedallowedallowed–
8384dootchi.aiNo robots.txtallowedallowedallowedallowedallowedallowedallowedallowed–
8572openart.aiOpenallowedallowedallowedallowedallowedallowedallowedallowed–
8654openclaw.aiOpenallowedallowedallowedallowedallowedallowedallowedallowedyes
8772kling.aiOpenallowedpartialpartialallowedpartialpartialpartialallowedyes
8895ctrltech.aiOpenallowedallowedallowedallowedallowedallowedallowedallowed–
8954leonardo.aiOpenallowedallowedallowedallowedallowedallowedallowedallowed–
9117devin.aiOpenpartialpartialpartialpartialpartialpartialpartialallowedyes
9121dimensions.aiOpenallowedallowedallowedallowedallowedallowedallowedallowed–
9523lmstudio.aiOpenallowedallowedallowedallowedallowedallowedallowedallowedyes
9563datahive.aiOpenallowedallowedallowedallowedallowedallowedallowedallowed–
9657paradox.aiOpenallowedallowedallowedallowedallowedallowedallowedallowedyes
9779eightfold.aiOpenallowedallowedallowedallowedallowedallowedallowedallowedyes
9786yunray.aiUnreadableunknownunknownunknownunknownunknownunknownunknownunknown–

Categories on .ai

FAQ

Do .ai websites block AI crawlers?
As of 2026-10-02, 4.8% of the 62 .ai websites whose robots.txt we could read block GPTBot, 3.2% block ClaudeBot and 1.6% block PerplexityBot (all 8,366 sites: 12%, 11%, 8.3%). 26 of them (41.9%) serve an llms.txt (all sites: 15.2%).
Which .ai websites block the most AI crawlers?
mgln.ai (29 of 29), claude.ai (4 of 29), tripo3d.ai (3 of 29), scite.ai (2 of 29). The full per-site status is on each site page.
How is this measured?
We fetch each site's robots.txt every day from one residential IP in Taiwan with an honest user agent, and apply the rules to 29 AI crawlers for the home page. 64 .ai sites are in the sample, 62 had a readable robots.txt (2 did not answer or returned an error page).

Other countries and suffixes

Compare all suffixes