TidyToolsAI Crawler Index · data as of · updated daily

AI Crawler Index · country

Do websites in Iran (.ir) block AI crawlers?

As of 2026-10-03, 0% of the 35 websites in Iran (.ir) whose robots.txt we could read block GPTBot, 0% block ClaudeBot and 0% block PerplexityBot (all 8,357 sites: 12%, 11%, 8.3%). 4 of them (11.4%) serve an llms.txt (all sites: 15.2%).

0%
block GPTBot (all sites 12%)
0%
block ClaudeBot (all sites 11%)
11.4%
serve an llms.txt (all sites 15.2%)
35
of 39 sites with a readable robots.txt

Block rate by crawler: .ir (Iran) vs all sites

CrawlerOperator.ir (Iran)All sitesDifference
AmazonbotAmazon0%9.2%-9.2 pp
Amzn-SearchBotAmazon0%4.1%-4.1 pp
Amzn-UserAmazon0%3.9%-3.9 pp
ApplebotApple0%3.1%-3.1 pp
Applebot-ExtendedApple0%9.6%-9.6 pp
BytespiderByteDance0%12.3%-12.3 pp
CCBotCommon Crawl0%12.8%-12.8 pp
ChatGPT-UserOpenAI0%7.5%-7.5 pp
Claude-SearchBotAnthropic0%5.4%-5.4 pp
Claude-UserAnthropic0%5.5%-5.5 pp
ClaudeBotAnthropic0%11%-11 pp
cohere-aiCohere0%9.1%-9.1 pp
DuckAssistBotDuckDuckGo0%5.8%-5.8 pp
Google-AgentGoogle0%3.2%-3.2 pp
Google-CloudVertexBotGoogle0%5.2%-5.2 pp
Google-ExtendedGoogle0%9.5%-9.5 pp
Google-GeminiNotebookGoogle0%3%-3 pp
GPTBotOpenAI0%12%-12 pp
meta-externaladsMeta0%2.8%-2.8 pp
Meta-ExternalAgentMeta0%9.5%-9.5 pp
meta-externalfetcherMeta0%6%-6 pp
meta-webindexerMeta0%4.6%-4.6 pp
MistralAI-IndexMistral0%3.3%-3.3 pp
MistralAI-TrainingMistral0%3.3%-3.3 pp
MistralAI-UserMistral0%5.4%-5.4 pp
OAI-AdsBotOpenAI0%3%-3 pp
OAI-SearchBotOpenAI0%5.7%-5.7 pp
Perplexity-UserPerplexity0%5.7%-5.7 pp
PerplexityBotPerplexity0%8.3%-8.3 pp

Policy mix

PolicySitesShare
Open3282.1%
Unreadable410.3%
No robots.txt37.7%

Most restrictive .ir (Iran) sites

No .ir (Iran) site in the index blocks any of the 29 crawlers on its home page.

Sites with an llms.txt (4)

All 39 .ir (Iran) sites

✕ blocked · ◐ partially blocked (some paths) · ✓ allowed · ? robots.txt unreadable. "#" is the position in our popularity-ordered sample.

#SitePolicyGPTBotOAI-SearchChatGPT-UserClaudeBotClaude-SearchPerplexityGoogle-Ext.CCBotllms.txt
582divar.irOpenallowedallowedallowedallowedallowedallowedallowedallowed–
741musicdel.irOpenallowedallowedallowedallowedallowedallowedallowedallowedyes
742iau.irUnreadableunknownunknownunknownunknownunknownunknownunknownunknown–
974easytrader.irNo robots.txtallowedallowedallowedallowedallowedallowedallowedallowed–
2793ble.irOpenallowedallowedallowedallowedallowedallowedallowedallowed–
3257irna.irOpenallowedallowedallowedallowedallowedallowedallowedallowed–
3677farsnews.irOpenallowedallowedallowedallowedallowedallowedallowedallowed–
4057isna.irNo robots.txtallowedallowedallowedallowedallowedallowedallowedallowed–
4121rubika.irNo robots.txtallowedallowedallowedallowedallowedallowedallowedallowed–
4278snapp.irUnreadableunknownunknownunknownunknownunknownunknownunknownunknown–
4481irancell.irOpenallowedallowedallowedallowedallowedallowedallowedallowed–
4502khabaronline.irOpenallowedallowedallowedallowedallowedallowedallowedallowed–
4653tabnak.irOpenallowedallowedallowedallowedallowedallowedallowedallowed–
4831zoomit.irOpenallowedallowedallowedallowedallowedallowedallowedallowed–
5079tamin.irUnreadableunknownunknownunknownunknownunknownunknownunknownunknown–
5344abadis.irOpenallowedallowedallowedallowedallowedallowedallowedallowed–
5643bahesab.irOpenallowedallowedallowedallowedallowedallowedallowedallowed–
5661nobat.irOpenallowedallowedallowedallowedallowedallowedallowedallowedyes
5666taw-bio.irOpenallowedallowedallowedallowedallowedallowedallowedallowed–
5732tala.irOpenallowedallowedallowedallowedallowedallowedallowedallowed–
5810musicviral.irOpenallowedallowedallowedallowedallowedallowedallowedallowed–
5814melody98.irOpenallowedallowedallowedallowedallowedallowedallowedallowed–
5817musicsweb.irOpenallowedallowedallowedallowedallowedallowedallowedallowed–
5908iranjib.irOpenallowedallowedallowedallowedallowedallowedallowedallowed–
5964kanoon.irOpenallowedallowedallowedallowedallowedallowedallowedallowed–
5999biomaze.irUnreadableunknownunknownunknownunknownunknownunknownunknownunknown–
6062uptune.irOpenallowedallowedallowedallowedallowedallowedallowedallowed–
6164bahalmag.irOpenallowedallowedallowedallowedallowedallowedallowedallowed–
6282abzarek.irOpenallowedallowedallowedallowedallowedallowedallowedallowed–
6322mt98.irOpenallowedallowedallowedallowedallowedallowedallowedallowed–
6685bama.irOpenallowedallowedallowedallowedallowedallowedallowedallowed–
7037fardayetaze.irOpenallowedallowedallowedallowedallowedallowedallowedallowedyes
7443cafebazaar.irOpenallowedallowedallowedallowedallowedallowedallowedallowed–
8697tasnimnews.irOpenallowedallowedallowedallowedallowedallowedallowedallowed–
9052telewebion.irOpenallowedallowedallowedallowedallowedallowedallowedallowed–
9211metrix.irOpenallowedallowedallowedallowedallowedallowedallowedallowed–
9484hamshahrionline.irOpenallowedallowedallowedallowedallowedallowedallowedallowed–
9663myket.irOpenallowedallowedallowedallowedallowedallowedallowedallowedyes
9938entekhab.irOpenallowedallowedallowedallowedallowedallowedallowedallowed–

Categories on .ir (Iran)

FAQ

Do websites in Iran (.ir) block AI crawlers?
As of 2026-10-03, 0% of the 35 websites in Iran (.ir) whose robots.txt we could read block GPTBot, 0% block ClaudeBot and 0% block PerplexityBot (all 8,357 sites: 12%, 11%, 8.3%). 4 of them (11.4%) serve an llms.txt (all sites: 15.2%).
Which .ir (Iran) websites block the most AI crawlers?
None of the 35 readable .ir (Iran) sites blocks an AI crawler on its home page.
How is this measured?
We fetch each site's robots.txt every day from one residential IP in Taiwan with an honest user agent, and apply the rules to 29 AI crawlers for the home page. 39 .ir (Iran) sites are in the sample, 35 had a readable robots.txt (4 did not answer or returned an error page).

Other countries and suffixes

Compare all suffixes