TidyToolsAI Crawler Index · data as of · updated daily

AI Crawler Index · domain suffix

Do .app websites block AI crawlers?

As of 2026-10-03, 16.7% of the 30 .app websites whose robots.txt we could read block GPTBot, 16.7% block ClaudeBot and 13.3% block PerplexityBot (all 8,357 sites: 12%, 11%, 8.3%). 4 of them (13.3%) serve an llms.txt (all sites: 15.2%).

16.7%
block GPTBot (all sites 12%)
16.7%
block ClaudeBot (all sites 11%)
13.3%
serve an llms.txt (all sites 15.2%)
30
of 31 sites with a readable robots.txt

Block rate by crawler: .app vs all sites

CrawlerOperator.appAll sitesDifference
AmazonbotAmazon16.7%9.2%+7.5 pp
Amzn-UserAmazon16.7%3.9%+12.8 pp
Applebot-ExtendedApple16.7%9.6%+7.1 pp
BytespiderByteDance16.7%12.3%+4.4 pp
CCBotCommon Crawl16.7%12.8%+3.9 pp
ChatGPT-UserOpenAI16.7%7.5%+9.2 pp
Claude-UserAnthropic16.7%5.5%+11.2 pp
ClaudeBotAnthropic16.7%11%+5.7 pp
cohere-aiCohere16.7%9.1%+7.6 pp
Google-AgentGoogle16.7%3.2%+13.5 pp
Google-CloudVertexBotGoogle16.7%5.2%+11.5 pp
Google-ExtendedGoogle16.7%9.5%+7.2 pp
Google-GeminiNotebookGoogle16.7%3%+13.7 pp
GPTBotOpenAI16.7%12%+4.7 pp
Meta-ExternalAgentMeta16.7%9.5%+7.2 pp
meta-externalfetcherMeta16.7%6%+10.7 pp
MistralAI-TrainingMistral16.7%3.3%+13.4 pp
MistralAI-UserMistral16.7%5.4%+11.3 pp
Perplexity-UserPerplexity16.7%5.7%+11 pp
Amzn-SearchBotAmazon13.3%4.1%+9.2 pp
ApplebotApple13.3%3.1%+10.2 pp
Claude-SearchBotAnthropic13.3%5.4%+7.9 pp
DuckAssistBotDuckDuckGo13.3%5.8%+7.5 pp
meta-externaladsMeta13.3%2.8%+10.5 pp
meta-webindexerMeta13.3%4.6%+8.7 pp
MistralAI-IndexMistral13.3%3.3%+10 pp
OAI-AdsBotOpenAI13.3%3%+10.3 pp
OAI-SearchBotOpenAI13.3%5.7%+7.6 pp
PerplexityBotPerplexity13.3%8.3%+5 pp

Who blocks what, by name: GPTBot, ClaudeBot, PerplexityBot, Google-Extended, CCBot, Bytespider, Applebot-Extended, OAI-SearchBot.

Policy mix

PolicySitesShare
Open2064.5%
No robots.txt516.1%
Blocks all AI search412.9%
Unreadable13.2%
Blocks some AI search13.2%

Most restrictive .app sites

  1. sangtacviet.app 29 of 29 blocked
  2. okolo.app 29 of 29 blocked
  3. lucid.app 29 of 29 blocked
  4. harmonicsecurity.app 29 of 29 blocked
  5. animedekho.app 19 of 29 blocked

Sites with an llms.txt (4)

All 31 .app sites

✕ blocked · ◐ partially blocked (some paths) · ✓ allowed · ? robots.txt unreadable. "#" is the position in our popularity-ordered sample.

#SitePolicyGPTBotOAI-SearchChatGPT-UserClaudeBotClaude-SearchPerplexityGoogle-Ext.CCBotllms.txt
173bsky.appOpenallowedallowedallowedallowedallowedallowedallowedallowed–
259shop.appOpenallowedallowedallowedallowedallowedallowedallowedallowedyes
1309railway.appOpenallowedallowedallowedallowedallowedallowedallowedallowedyes
1405phantom.appOpenallowedallowedallowedallowedallowedallowedallowedallowed–
2123gamma.appUnreadableunknownunknownunknownunknownunknownunknownunknownunknown–
29562giskz.appOpenallowedallowedallowedallowedallowedallowedallowedallowed–
3947keplr.appOpenallowedallowedallowedallowedallowedallowedallowedallowed–
4516gapgpt.appOpenpartialallowedpartialallowedallowedallowedallowedallowed–
4706cash.appOpenallowedallowedallowedallowedallowedallowedallowedallowed–
5039movieboxpro.appOpenallowedallowedallowedallowedallowedallowedallowedallowed–
5084saveclip.appOpenallowedallowedallowedallowedallowedallowedallowedallowed–
5460snapsave.appOpenallowedallowedallowedallowedallowedallowedallowedallowed–
6154linear.appOpenallowedallowedallowedallowedallowedallowedallowedallowedyes
6186animedekho.appBlocks some AI searchblockedallowedblockedblockedallowedallowedblockedblocked–
6248oasisnet.appOpenallowedallowedallowedallowedallowedallowedallowedallowed–
6251sangtacviet.appBlocks all AI searchblockedblockedblockedblockedblockedblockedblockedblocked–
6692nodr.appOpenallowedallowedallowedallowedallowedallowedallowedallowed–
6931sosalkino.appOpenallowedallowedallowedallowedallowedallowedallowedallowed–
6979s-gift.appNo robots.txtallowedallowedallowedallowedallowedallowedallowedallowed–
70357tv.appNo robots.txtallowedallowedallowedallowedallowedallowedallowedallowed–
7094ngrok-free.appNo robots.txtallowedallowedallowedallowedallowedallowedallowedallowed–
7141parsec.appOpenallowedallowedallowedallowedallowedallowedallowedallowed–
7759ajay.appNo robots.txtallowedallowedallowedallowedallowedallowedallowedallowed–
8054knock.appOpenallowedallowedallowedallowedallowedallowedallowedallowedyes
8672base44.appOpenallowedallowedallowedallowedallowedallowedallowedallowed–
9114premid.appOpenallowedallowedallowedallowedallowedallowedallowedallowed–
9123okolo.appBlocks all AI searchblockedblockedblockedblockedblockedblockedblockedblocked–
9385lucid.appBlocks all AI searchblockedblockedblockedblockedblockedblockedblockedblocked–
9510harmonicsecurity.appBlocks all AI searchblockedblockedblockedblockedblockedblockedblockedblocked–
9692cinemahq.appOpenallowedallowedallowedallowedallowedallowedallowedallowed–
9715loilonote.appNo robots.txtallowedallowedallowedallowedallowedallowedallowedallowed–

Categories on .app

FAQ

Do .app websites block AI crawlers?
As of 2026-10-03, 16.7% of the 30 .app websites whose robots.txt we could read block GPTBot, 16.7% block ClaudeBot and 13.3% block PerplexityBot (all 8,357 sites: 12%, 11%, 8.3%). 4 of them (13.3%) serve an llms.txt (all sites: 15.2%).
Which .app websites block the most AI crawlers?
sangtacviet.app (29 of 29), okolo.app (29 of 29), lucid.app (29 of 29), harmonicsecurity.app (29 of 29), animedekho.app (19 of 29). The full per-site status is on each site page.
How is this measured?
We fetch each site's robots.txt every day from one residential IP in Taiwan with an honest user agent, and apply the rules to 29 AI crawlers for the home page. 31 .app sites are in the sample, 30 had a readable robots.txt (1 did not answer or returned an error page).

Other countries and suffixes

Compare all suffixes