TidyToolsAI Crawler Index · data as of · updated daily

AI Crawler Index · domain suffix

Do .net websites block AI crawlers?

As of 2026-10-03, 13.6% of the 337 .net websites whose robots.txt we could read block GPTBot, 13.6% block ClaudeBot and 10.4% block PerplexityBot (all 8,357 sites: 12%, 11%, 8.3%). 31 of them (9.2%) serve an llms.txt (all sites: 15.2%).

13.6%
block GPTBot (all sites 12%)
13.6%
block ClaudeBot (all sites 11%)
9.2%
serve an llms.txt (all sites 15.2%)
337
of 409 sites with a readable robots.txt

Block rate by crawler: .net vs all sites

CrawlerOperator.netAll sitesDifference
CCBotCommon Crawl13.9%12.8%+1.1 pp
ClaudeBotAnthropic13.6%11%+2.6 pp
GPTBotOpenAI13.6%12%+1.6 pp
BytespiderByteDance12.8%12.3%+0.5 pp
Google-ExtendedGoogle11.9%9.5%+2.4 pp
Meta-ExternalAgentMeta11.3%9.5%+1.8 pp
Applebot-ExtendedApple11%9.6%+1.4 pp
cohere-aiCohere10.7%9.1%+1.6 pp
PerplexityBotPerplexity10.4%8.3%+2.1 pp
ChatGPT-UserOpenAI10.1%7.5%+2.6 pp
AmazonbotAmazon9.8%9.2%+0.6 pp
Claude-UserAnthropic8%5.5%+2.5 pp
meta-externalfetcherMeta8%6%+2 pp
OAI-SearchBotOpenAI8%5.7%+2.3 pp
Claude-SearchBotAnthropic7.7%5.4%+2.3 pp
Perplexity-UserPerplexity7.7%5.7%+2 pp
Google-CloudVertexBotGoogle7.4%5.2%+2.2 pp
DuckAssistBotDuckDuckGo7.1%5.8%+1.3 pp
MistralAI-TrainingMistral7.1%3.3%+3.8 pp
MistralAI-UserMistral7.1%5.4%+1.7 pp
meta-webindexerMeta6.8%4.6%+2.2 pp
Amzn-SearchBotAmazon6.5%4.1%+2.4 pp
Amzn-UserAmazon6.5%3.9%+2.6 pp
Google-AgentGoogle6.5%3.2%+3.3 pp
Google-GeminiNotebookGoogle6.5%3%+3.5 pp
OAI-AdsBotOpenAI6.5%3%+3.5 pp
ApplebotApple6.2%3.1%+3.1 pp
meta-externaladsMeta6.2%2.8%+3.4 pp
MistralAI-IndexMistral6.2%3.3%+2.9 pp

Who blocks what, by name: GPTBot, ClaudeBot, PerplexityBot, Google-Extended, CCBot, Bytespider, Applebot-Extended, OAI-SearchBot.

Policy mix

PolicySitesShare
Open19948.7%
No robots.txt7819.1%
Unreadable7217.6%
Blocks some AI search235.6%
Blocks all AI search194.6%
Blocks training only184.4%

Most restrictive .net sites

  1. expressapisv2.net 29 of 29 blocked
  2. wargaming.net 29 of 29 blocked
  3. rs6.net 29 of 29 blocked
  4. entrust.net 29 of 29 blocked
  5. cnki.net 29 of 29 blocked
  6. myhostadmin.net 29 of 29 blocked
  7. paycomonline.net 29 of 29 blocked
  8. storygize.net 29 of 29 blocked
  9. islonline.net 29 of 29 blocked
  10. widen.net 29 of 29 blocked
  11. 13o.net 29 of 29 blocked
  12. fonts.net 29 of 29 blocked
  13. doujin-lc.net 29 of 29 blocked
  14. parspack.net 29 of 29 blocked
  15. zdbb.net 29 of 29 blocked

Sites with an llms.txt (31)

Top 100 of 409 .net sites

✕ blocked · ◐ partially blocked (some paths) · ✓ allowed · ? robots.txt unreadable. "#" is the position in our popularity-ordered sample.

#SitePolicyGPTBotOAI-SearchChatGPT-UserClaudeBotClaude-SearchPerplexityGoogle-Ext.CCBotllms.txt
20gandi.netOpenallowedallowedallowedallowedallowedallowedallowedallowed–
89telekom.netNo robots.txtallowedallowedallowedallowedallowedallowedallowedallowed–
99researchgate.netOpenallowedallowedallowedallowedallowedallowedallowedallowed–
102comcast.netUnreadableunknownunknownunknownunknownunknownunknownunknownunknown–
107sourceforge.netOpenallowedallowedallowedallowedallowedallowedallowedallowed–
108cpanel.netOpenallowedallowedallowedallowedallowedallowedallowedallowed–
117launchpad.netBlocks some AI searchblockedblockedallowedblockedblockedblockedblockedblocked–
140php.netOpenallowedallowedallowedallowedallowedallowedallowedallowed–
204speedtest.netOpenallowedallowedallowedallowedallowedallowedallowedallowed–
219bitdefender.netNo robots.txtallowedallowedallowedallowedallowedallowedallowedallowed–
225behance.netBlocks training onlyblockedallowedallowedblockedallowedallowedblockedblocked–
234slideshare.netBlocks training onlyallowedallowedallowedblockedallowedallowedallowedblocked–
271ovh.netUnreadableunknownunknownunknownunknownunknownunknownunknownunknown–
384daum.netOpenallowedallowedallowedallowedallowedallowedallowedallowed–
441pixiv.netBlocks training onlyblockedallowedallowedblockedallowedallowedblockedblocked–
477themeforest.netBlocks training onlyallowedallowedallowedallowedallowedallowedallowedblocked–
546inleed.netNo robots.txtallowedallowedallowedallowedallowedallowedallowedallowed–
576savefrom.netOpenallowedallowedallowedallowedallowedallowedallowedallowedyes
599fanfiction.netUnreadableunknownunknownunknownunknownunknownunknownunknownunknown–
614battle.netUnreadableunknownunknownunknownunknownunknownunknownunknownunknown–
634calculator.netOpenallowedallowedallowedallowedallowedallowedallowedallowed–
636jalan.netOpenallowedallowedallowedallowedallowedallowedallowedallowed–
639hollywoodbets.netUnreadableunknownunknownunknownunknownunknownunknownunknownunknown–
652fdown.netOpenallowedallowedallowedallowedallowedallowedallowedallowed–
672ficbook.netOpenallowedallowedallowedallowedallowedallowedallowedallowed–
676carsensor.netOpenallowedallowedallowedallowedallowedallowedallowedallowed–
685zedge.netBlocks training onlyblockedallowedallowedallowedallowedallowedallowedallowed–
719eenadu.netUnreadableunknownunknownunknownunknownunknownunknownunknownunknown–
730dogdrip.netOpenallowedallowedallowedallowedallowedallowedallowedallowed–
738programme-tv.netBlocks training onlyallowedallowedallowedallowedallowedallowedallowedblockedyes
761therapservices.netOpenpartialpartialpartialpartialallowedpartialpartialpartialyes
764wordwall.netOpenallowedallowedallowedallowedallowedallowedallowedallowed–
766theqoo.netUnreadableunknownunknownunknownunknownunknownunknownunknownunknown–
775girlschannel.netOpenallowedallowedallowedallowedallowedallowedallowedallowed–
776czbooks.netNo robots.txtallowedallowedallowedallowedallowedallowedallowedallowed–
916liquipedia.netBlocks some AI searchblockedallowedblockedblockedallowedallowedallowedallowed–
937appier.netOpenallowedallowedallowedallowedallowedallowedallowedallowed–
985desikahani2.netOpenallowedallowedallowedallowedallowedallowedallowedallowed–
1020charter.netOpenallowedallowedallowedallowedallowedallowedallowedallowed–
1029iponweb.netNo robots.txtallowedallowedallowedallowedallowedallowedallowedallowed–
1033earthlink.netOpenallowedallowedallowedallowedallowedallowedallowedallowed–
1099av380.netNo robots.txtallowedallowedallowedallowedallowedallowedallowedallowed–
1100virtualearth.netNo robots.txtallowedallowedallowedallowedallowedallowedallowedallowed–
1177mintegral.netOpenallowedallowedallowedallowedallowedallowedallowedallowed–
1210bunny.netOpenallowedallowedallowedallowedallowedallowedallowedallowedyes
1226pubnative.netOpenallowedallowedallowedallowedallowedallowedallowedallowedyes
1240smi2.netOpenallowedallowedallowedallowedallowedallowedallowedallowed–
1289gmx.netOpenpartialpartialpartialpartialpartialpartialpartialpartial–
1344churnzero.netOpenallowedallowedallowedallowedallowedallowedallowedallowed–
1345expressapisv2.netBlocks all AI searchblockedblockedblockedblockedblockedblockedblockedblocked–
1477sensic.netNo robots.txtallowedallowedallowedallowedallowedallowedallowedallowed–
1497trafficjunky.netNo robots.txtallowedallowedallowedallowedallowedallowedallowedallowed–
1505faz.netBlocks some AI searchblockedblockedblockedblockedblockedblockedblockedblocked–
1596a1.netOpenallowedallowedallowedallowedallowedallowedallowedallowed–
1616w55c.netUnreadableunknownunknownunknownunknownunknownunknownunknownunknown–
1671c6gj-static.netOpenpartialallowedallowedallowedallowedpartialpartialallowedyes
1701myanimelist.netBlocks some AI searchblockedallowedallowedblockedallowedallowedblockedblocked–
1726gaijin.netOpenallowedallowedallowedallowedallowedallowedallowedallowed–
1746uk.netOpenallowedallowedallowedallowedallowedallowedallowedallowed–
1748infojobs.netBlocks some AI searchblockedblockedblockedblockedblockedblockedblockedblocked–
1749wargaming.netBlocks all AI searchblockedblockedblockedblockedblockedblockedblockedblocked–
1778hinet.netNo robots.txtallowedallowedallowedallowedallowedallowedallowedallowed–
1781weatherbug.netUnreadableunknownunknownunknownunknownunknownunknownunknownunknown–
1793signalr.netNo robots.txtallowedallowedallowedallowedallowedallowedallowedallowed–
1813ripe.netOpenallowedallowedallowedallowedallowedallowedallowedallowed–
1836trouble-free.netNo robots.txtallowedallowedallowedallowedallowedallowedallowedallowed–
1837imgix.netUnreadableunknownunknownunknownunknownunknownunknownunknownunknown–
1868afrinic.netOpenallowedallowedallowedallowedallowedallowedallowedallowed–
1872reverso.netUnreadableunknownunknownunknownunknownunknownunknownunknownunknown–
1882mailgun.netOpenallowedallowedallowedallowedallowedallowedallowedallowed–
1898serverdata.netOpenallowedallowedallowedallowedallowedallowedallowedallowed–
1901emarsys.netUnreadableunknownunknownunknownunknownunknownunknownunknownunknown–
1928shawcable.netUnreadableunknownunknownunknownunknownunknownunknownunknownunknown–
1945handle.netOpenallowedallowedallowedallowedallowedallowedallowedallowed–
1954rs6.netBlocks all AI searchblockedblockedblockedblockedblockedblockedblockedblocked–
1963atl-paas.netNo robots.txtallowedallowedallowedallowedallowedallowedallowedallowed–
1970consentmanager.netOpenallowedallowedallowedallowedallowedallowedallowedallowed–
198417track.netUnreadableunknownunknownunknownunknownunknownunknownunknownunknown–
2015entrust.netBlocks all AI searchblockedblockedblockedblockedblockedblockedblockedblocked–
2087apnic.netOpenallowedallowedallowedallowedallowedallowedallowedallowed–
2128cox.netUnreadableunknownunknownunknownunknownunknownunknownunknownunknown–
2165footmercato.netOpenallowedallowedallowedallowedallowedallowedallowedallowed–
2178quad9.netOpenallowedallowedallowedallowedallowedallowedallowedallowed–
2220uuidksinc.netUnreadableunknownunknownunknownunknownunknownunknownunknownunknown–
2278gldn.netUnreadableunknownunknownunknownunknownunknownunknownunknownunknown–
2340zlib.netOpenallowedallowedallowedallowedallowedallowedallowedallowed–
2347avto.netUnreadableunknownunknownunknownunknownunknownunknownunknownunknown–
2353artfight.netBlocks some AI searchblockedallowedblockedblockedallowedblockedblockedblocked–
2363localto.netNo robots.txtallowedallowedallowedallowedallowedallowedallowedallowed–
2392thunderbird.netNo robots.txtallowedallowedallowedallowedallowedallowedallowedallowed–
2395cnki.netBlocks all AI searchblockedblockedblockedblockedblockedblockedblockedblocked–
2398desiviral.netOpenallowedallowedallowedallowedallowedallowedallowedallowedyes
2410skyscanner.netBlocks training onlyallowedallowedallowedallowedallowedallowedallowedallowed–
2418lectormangass.netBlocks some AI searchblockedallowedallowedblockedallowedblockedblockedblocked–
2452cookieadmin.netOpenallowedallowedallowedallowedallowedallowedallowedallowed–
2475myhostadmin.netBlocks all AI searchblockedblockedblockedblockedblockedblockedblockedblocked–
2480cableone.netUnreadableunknownunknownunknownunknownunknownunknownunknownunknown–
2485alarabiya.netUnreadableunknownunknownunknownunknownunknownunknownunknownunknown–
2499paycomonline.netBlocks all AI searchblockedblockedblockedblockedblockedblockedblockedblocked–
2507arin.netOpenallowedallowedallowedallowedallowedallowedallowedallowed–

Sorted by popularity; the rest are on the A-Z lists.

Categories on .net

FAQ

Do .net websites block AI crawlers?
As of 2026-10-03, 13.6% of the 337 .net websites whose robots.txt we could read block GPTBot, 13.6% block ClaudeBot and 10.4% block PerplexityBot (all 8,357 sites: 12%, 11%, 8.3%). 31 of them (9.2%) serve an llms.txt (all sites: 15.2%).
Which .net websites block the most AI crawlers?
expressapisv2.net (29 of 29), wargaming.net (29 of 29), rs6.net (29 of 29), entrust.net (29 of 29), cnki.net (29 of 29). The full per-site status is on each site page.
How is this measured?
We fetch each site's robots.txt every day from one residential IP in Taiwan with an honest user agent, and apply the rules to 29 AI crawlers for the home page. 409 .net sites are in the sample, 337 had a readable robots.txt (72 did not answer or returned an error page).

Other countries and suffixes

Compare all suffixes