TidyToolsAI Crawler Index · data as of · updated daily

AI Crawler Index · country

Do websites in Russia (.ru) block AI crawlers?

As of 2026-10-02, 5.1% of the 295 websites in Russia (.ru) whose robots.txt we could read block GPTBot, 2% block ClaudeBot and 1.4% block PerplexityBot (all 8,366 sites: 12%, 11%, 8.3%). 29 of them (9.8%) serve an llms.txt (all sites: 15.2%).

5.1%
block GPTBot (all sites 12%)
2%
block ClaudeBot (all sites 11%)
9.8%
serve an llms.txt (all sites 15.2%)
295
of 346 sites with a readable robots.txt

Block rate by crawler: .ru (Russia) vs all sites

CrawlerOperator.ru (Russia)All sitesDifference
GPTBotOpenAI5.1%12%-6.9 pp
BytespiderByteDance3.1%12.2%-9.1 pp
AmazonbotAmazon2.4%9.2%-6.8 pp
CCBotCommon Crawl2.4%12.7%-10.3 pp
ChatGPT-UserOpenAI2.4%7.4%-5 pp
ClaudeBotAnthropic2%11%-9 pp
cohere-aiCohere2%9%-7 pp
Meta-ExternalAgentMeta2%9.5%-7.5 pp
Amzn-SearchBotAmazon1.4%4.1%-2.7 pp
Applebot-ExtendedApple1.4%9.6%-8.2 pp
Google-ExtendedGoogle1.4%9.5%-8.1 pp
OAI-SearchBotOpenAI1.4%5.7%-4.3 pp
PerplexityBotPerplexity1.4%8.3%-6.9 pp
Amzn-UserAmazon1%3.9%-2.9 pp
ApplebotApple1%3.1%-2.1 pp
Claude-SearchBotAnthropic1%5.4%-4.4 pp
Claude-UserAnthropic1%5.5%-4.5 pp
DuckAssistBotDuckDuckGo1%5.7%-4.7 pp
Google-AgentGoogle1%3.2%-2.2 pp
Google-CloudVertexBotGoogle1%5.2%-4.2 pp
Google-GeminiNotebookGoogle1%3%-2 pp
meta-externaladsMeta1%2.8%-1.8 pp
meta-externalfetcherMeta1%6%-5 pp
meta-webindexerMeta1%4.6%-3.6 pp
MistralAI-IndexMistral1%3.3%-2.3 pp
MistralAI-TrainingMistral1%3.2%-2.2 pp
MistralAI-UserMistral1%5.3%-4.3 pp
OAI-AdsBotOpenAI1%3%-2 pp
Perplexity-UserPerplexity1%5.7%-4.7 pp

Who blocks what, by name: GPTBot, ClaudeBot, PerplexityBot, Google-Extended, CCBot, Bytespider, Applebot-Extended, OAI-SearchBot.

Policy mix

PolicySitesShare
Open23267.1%
Unreadable5114.7%
No robots.txt4613.3%
Blocks training only92.6%
Blocks some AI search51.4%
Blocks all AI search30.9%

Most restrictive .ru (Russia) sites

  1. taxtelecom.ru 29 of 29 blocked
  2. academic.ru 29 of 29 blocked
  3. komus.ru 29 of 29 blocked
  4. litmarket.ru 8 of 29 blocked
  5. sefan.ru 8 of 29 blocked
  6. mos.ru 5 of 29 blocked
  7. macomnet.ru 5 of 29 blocked
  8. hibiny.ru 4 of 29 blocked
  9. loveplanet.ru 3 of 29 blocked
  10. film.ru 3 of 29 blocked
  11. fontanka.ru 2 of 29 blocked
  12. 1c.ru 1 of 29 blocked
  13. hse.ru 1 of 29 blocked
  14. rg.ru 1 of 29 blocked
  15. klerk.ru 1 of 29 blocked

Sites with an llms.txt (29)

Top 100 of 346 .ru (Russia) sites

✕ blocked · ◐ partially blocked (some paths) · ✓ allowed · ? robots.txt unreadable. "#" is the position in our popularity-ordered sample.

#SitePolicyGPTBotOAI-SearchChatGPT-UserClaudeBotClaude-SearchPerplexityGoogle-Ext.CCBotllms.txt
8mail.ruOpenallowedallowedallowedallowedallowedallowedallowedallowed–
10dzen.ruOpenallowedallowedallowedallowedallowedallowedallowedallowed–
43yandex.ruOpenallowedpartialallowedallowedallowedallowedallowedallowed–
54reg.ruNo robots.txtallowedallowedallowedallowedallowedallowedallowedallowed–
69mts.ruOpenallowedallowedallowedallowedallowedallowedallowedallowed–
74vk.ruOpenallowedallowedallowedallowedallowedallowedallowedallowed–
101ozon.ruUnreadableunknownunknownunknownunknownunknownunknownunknownunknown–
131ok.ruOpenallowedallowedallowedallowedallowedallowedallowedallowed–
138drom.ruOpenallowedallowedallowedallowedallowedallowedallowedallowed–
139crpt.ruOpenallowedallowedallowedallowedallowedallowedallowedallowed–
155selectel.ruOpenallowedallowedallowedallowedallowedallowedallowedallowedyes
167netangels.ruOpenallowedallowedallowedallowedallowedallowedallowedallowed–
179timeweb.ruNo robots.txtallowedallowedallowedallowedallowedallowedallowedallowed–
217ivi.ruOpenallowedallowedallowedallowedallowedallowedallowedallowed–
220gosuslugi.ruOpenallowedallowedallowedallowedallowedallowedallowedallowed–
235wildberries.ruNo robots.txtallowedallowedallowedallowedallowedallowedallowedallowed–
236bitrix24.ruOpenpartialpartialpartialpartialpartialpartialpartialallowedyes
269avito.ruOpenallowedallowedallowedallowedallowedallowedallowedallowed–
304vkontakte.ruOpenallowedallowedallowedallowedallowedallowedallowedallowed–
309consultant.ruOpenallowedallowedallowedallowedallowedallowedallowedallowed–
324ya.ruOpenallowedallowedallowedallowedallowedallowedallowedallowed–
337beget.ruOpenallowedallowedallowedallowedallowedallowedallowedallowed–
361rt.ruUnreadableunknownunknownunknownunknownunknownunknownunknownunknown–
381wb.ruNo robots.txtallowedallowedallowedallowedallowedallowedallowedallowed–
393pikabu.ruUnreadableunknownunknownunknownunknownunknownunknownunknownunknown–
472kontur.ruOpenallowedallowedallowedallowedallowedallowedallowedallowedyes
520rustore.ruOpenallowedallowedallowedallowedallowedallowedallowedallowed–
565tabor.ruOpenallowedallowedallowedallowedallowedallowedallowedallowed–
578rbc.ruOpenallowedallowedallowedallowedallowedallowedallowedallowed–
579rutube.ruOpenallowedallowedallowedallowedallowedallowedallowedallowed–
598beboo.ruOpenallowedallowedallowedallowedallowedallowedallowedallowed–
628edna.ruUnreadableunknownunknownunknownunknownunknownunknownunknownunknown–
635jino.ruOpenallowedallowedallowedallowedallowedallowedallowedallowed–
674mangabuff.ruOpenallowedallowedallowedallowedallowedallowedallowedallowed–
701max.ruOpenallowedallowedallowedallowedallowedallowedallowedallowed–
792rambler.ruOpenallowedallowedallowedallowedallowedallowedallowedallowed–
856amocrm.ruOpenallowedallowedallowedallowedallowedallowedallowedallowed–
858beeline.ruOpenallowedallowedallowedallowedallowedallowedallowedallowed–
866t2.ruUnreadableunknownunknownunknownunknownunknownunknownunknownunknown–
8801c.ruBlocks training onlyallowedallowedallowedallowedallowedallowedallowedallowed–
894sweb.ruOpenallowedallowedallowedallowedallowedallowedallowedallowed–
899gdemoideti.ruOpenallowedallowedallowedallowedallowedallowedallowedallowedyes
927sprinthost.ruOpenallowedallowedallowedallowedallowedallowedallowedallowed–
960mindbox.ruOpenallowedallowedallowedallowedallowedallowedallowedallowedyes
988hh.ruOpenpartialpartialpartialpartialallowedpartialpartialpartial–
1092vkvideo.ruOpenallowedallowedallowedallowedallowedallowedallowedallowed–
1102mirtesen.ruOpenallowedallowedallowedallowedallowedallowedallowedallowed–
1179ucoz.ruOpenallowedallowedallowedallowedallowedallowedallowedallowedyes
1186kinopoisk.ruOpenallowedallowedallowedallowedallowedallowedallowedallowed–
1189magnit.ruOpenallowedallowedallowedallowedallowedallowedallowedallowed–
1206ria.ruOpenallowedallowedallowedallowedallowedallowedallowedallowed–
1217123tt.ruUnreadableunknownunknownunknownunknownunknownunknownunknownunknown–
1244x5.ruOpenallowedallowedallowedallowedallowedallowedallowedallowed–
1248mos.ruBlocks some AI searchblockedallowedblockedallowedallowedallowedallowedblocked–
1310cdek.ruUnreadableunknownunknownunknownunknownunknownunknownunknownunknown–
1391tutu.ruOpenallowedallowedallowedallowedallowedallowedallowedallowedyes
1426ihc.ruOpenallowedallowedallowedallowedallowedallowedallowedallowedyes
1462apptracer.ruNo robots.txtallowedallowedallowedallowedallowedallowedallowedallowed–
1507sberdevices.ruOpenallowedallowedallowedallowedallowedallowedallowedallowed–
1515hostingradio.ruUnreadableunknownunknownunknownunknownunknownunknownunknownunknown–
1522narod.ruOpenallowedallowedallowedallowedallowedallowedallowedallowed–
1548raiffeisen.ruOpenallowedallowedallowedallowedallowedallowedallowedallowed–
1554profi.ruOpenallowedallowedallowedallowedallowedallowedallowedallowed–
1630firstvds.ruOpenallowedallowedallowedallowedallowedallowedallowedallowed–
1673sbis.ruNo robots.txtallowedallowedallowedallowedallowedallowedallowedallowed–
1683lenta.ruOpenallowedallowedallowedallowedallowedallowedallowedallowed–
1745saby.ruOpenallowedallowedallowedallowedallowedallowedallowedallowed–
1763ix.ruNo robots.txtallowedallowedallowedallowedallowedallowedallowedallowed–
1807vkusvill.ruUnreadableunknownunknownunknownunknownunknownunknownunknownunknown–
1817cbr.ruOpenallowedallowedallowedallowedallowedallowedallowedallowed–
1835likeevideo.ruOpenallowedallowedallowedallowedallowedallowedallowedallowed–
1865ktalk.ruUnreadableunknownunknownunknownunknownunknownunknownunknownunknown–
1867hse.ruBlocks training onlyblockedallowedallowedallowedallowedallowedallowedallowed–
1906linkis.ruNo robots.txtallowedallowedallowedallowedallowedallowedallowedallowed–
1958cian.ruUnreadableunknownunknownunknownunknownunknownunknownunknownunknown–
1974macomnet.ruBlocks some AI searchblockedallowedblockedallowedallowedallowedallowedblocked–
1977garant.ruUnreadableunknownunknownunknownunknownunknownunknownunknownunknown–
2009tass.ruUnreadableunknownunknownunknownunknownunknownunknownunknownunknown–
2036finam.ruUnreadableunknownunknownunknownunknownunknownunknownunknownunknown–
2075kp.ruOpenallowedallowedallowedallowedallowedallowedallowedallowed–
2083banki.ruNo robots.txtallowedallowedallowedallowedallowedallowedallowedallowed–
2088fotostrana.ruOpenallowedallowedallowedallowedallowedallowedallowedallowed–
2102vseinstrumenti.ruUnreadableunknownunknownunknownunknownunknownunknownunknownunknown–
2111goldapple.ruOpenallowedallowedallowedallowedallowedallowedallowedallowedyes
2119net.ruNo robots.txtallowedallowedallowedallowedallowedallowedallowedallowed–
2137sber.ruNo robots.txtallowedallowedallowedallowedallowedallowedallowedallowed–
2145cloud4box.ruOpenallowedallowedallowedallowedallowedallowedallowedallowedyes
2188abcp.ruOpenallowedallowedallowedallowedallowedallowedallowedallowedyes
2210aif.ruOpenallowedallowedallowedallowedallowedallowedallowedallowed–
2222samokat.ruUnreadableunknownunknownunknownunknownunknownunknownunknownunknown–
2229vkplay.ruOpenallowedallowedallowedallowedallowedallowedallowedallowed–
2232nag.ruNo robots.txtallowedallowedallowedallowedallowedallowedallowedallowed–
2248mk.ruOpenallowedallowedallowedallowedallowedallowedallowedallowed–
2268tw1.ruNo robots.txtallowedallowedallowedallowedallowedallowedallowedallowed–
2280prosv.ruOpenallowedallowedallowedallowedallowedallowedallowedallowed–
2282pochta.ruNo robots.txtallowedallowedallowedallowedallowedallowedallowedallowed–
2292org.ruNo robots.txtallowedallowedallowedallowedallowedallowedallowedallowed–
2293pp.ruNo robots.txtallowedallowedallowedallowedallowedallowedallowedallowed–
2319tns-counter.ruOpenallowedallowedallowedallowedallowedallowedallowedallowed–
2350playfamily.ruUnreadableunknownunknownunknownunknownunknownunknownunknownunknown–

Sorted by popularity; the rest are on the A-Z lists.

Categories on .ru (Russia)

FAQ

Do websites in Russia (.ru) block AI crawlers?
As of 2026-10-02, 5.1% of the 295 websites in Russia (.ru) whose robots.txt we could read block GPTBot, 2% block ClaudeBot and 1.4% block PerplexityBot (all 8,366 sites: 12%, 11%, 8.3%). 29 of them (9.8%) serve an llms.txt (all sites: 15.2%).
Which .ru (Russia) websites block the most AI crawlers?
taxtelecom.ru (29 of 29), academic.ru (29 of 29), komus.ru (29 of 29), litmarket.ru (8 of 29), sefan.ru (8 of 29). The full per-site status is on each site page.
How is this measured?
We fetch each site's robots.txt every day from one residential IP in Taiwan with an honest user agent, and apply the rules to 29 AI crawlers for the home page. 346 .ru (Russia) sites are in the sample, 295 had a readable robots.txt (51 did not answer or returned an error page).

Other countries and suffixes

Compare all suffixes