Which .in (India) websites block the most AI crawlers?

lnkd.in (26 of 29), npci.org.in (18 of 29), brainly.in (2 of 29), theprint.in (2 of 29). The full per-site status is on each site page.

How is this measured?

We fetch each site's robots.txt every day from one residential IP in Taiwan with an honest user agent, and apply the rules to 29 AI crawlers for the home page. 93 .in (India) sites are in the sample, 74 had a readable robots.txt (19 did not answer or returned an error page).

AI Crawler Index · country

Do websites in India (.in) block AI crawlers?

As of 2026-10-02, 4.1% of the 74 websites in India (.in) whose robots.txt we could read block GPTBot, 1.4% block ClaudeBot and 1.4% block PerplexityBot (all 8,366 sites: 12%, 11%, 8.3%). 8 of them (10.8%) serve an llms.txt (all sites: 15.2%).

4.1%

block GPTBot (all sites 12%)

1.4%

block ClaudeBot (all sites 11%)

10.8%

serve an llms.txt (all sites 15.2%)

of 93 sites with a readable robots.txt

Block rate by crawler: .in (India) vs all sites

Crawler	Operator	.in (India)	All sites	Difference
GPTBot	OpenAI	4.1%	12%	-7.9 pp
Amazonbot	Amazon	2.7%	9.2%	-6.5 pp
Amzn-SearchBot	Amazon	2.7%	4.1%	-1.4 pp
Amzn-User	Amazon	2.7%	3.9%	-1.2 pp
Applebot-Extended	Apple	2.7%	9.6%	-6.9 pp
Bytespider	ByteDance	2.7%	12.2%	-9.5 pp
CCBot	Common Crawl	2.7%	12.7%	-10 pp
Claude-User	Anthropic	2.7%	5.5%	-2.8 pp
cohere-ai	Cohere	2.7%	9%	-6.3 pp
DuckAssistBot	DuckDuckGo	2.7%	5.7%	-3 pp
Google-Agent	Google	2.7%	3.2%	-0.5 pp
Google-GeminiNotebook	Google	2.7%	3%	-0.3 pp
meta-externalads	Meta	2.7%	2.8%	-0.1 pp
meta-externalfetcher	Meta	2.7%	6%	-3.3 pp
meta-webindexer	Meta	2.7%	4.6%	-1.9 pp
MistralAI-Index	Mistral	2.7%	3.3%	-0.6 pp
MistralAI-Training	Mistral	2.7%	3.2%	-0.5 pp
MistralAI-User	Mistral	2.7%	5.3%	-2.6 pp
OAI-AdsBot	OpenAI	2.7%	3%	-0.3 pp
Perplexity-User	Perplexity	2.7%	5.7%	-3 pp
ChatGPT-User	OpenAI	1.4%	7.4%	-6 pp
Claude-SearchBot	Anthropic	1.4%	5.4%	-4 pp
ClaudeBot	Anthropic	1.4%	11%	-9.6 pp
Google-CloudVertexBot	Google	1.4%	5.2%	-3.8 pp
Google-Extended	Google	1.4%	9.5%	-8.1 pp
Meta-ExternalAgent	Meta	1.4%	9.5%	-8.1 pp
PerplexityBot	Perplexity	1.4%	8.3%	-6.9 pp
Applebot	Apple	0%	3.1%	-3.1 pp
OAI-SearchBot	OpenAI	0%	5.7%	-5.7 pp

Who blocks what, by name: GPTBot.

Policy mix

Policy	Sites	Share
Open	44	47.3%
No robots.txt	26	28%
Unreadable	19	20.4%
Blocks some AI search	2	2.2%
Blocks training only	2	2.2%

Most restrictive .in (India) sites

lnkd.in 26 of 29 blocked
npci.org.in 18 of 29 blocked
brainly.in 2 of 29 blocked
theprint.in 2 of 29 blocked

Sites with an llms.txt (8)

All 93 .in (India) sites

✕ blocked · ◐ partially blocked (some paths) · ✓ allowed · ? robots.txt unreadable. "#" is the position in our popularity-ordered sample.

#	Site	Policy	GPTBot	OAI-Search	ChatGPT-User	ClaudeBot	Claude-Search	Perplexity	Google-Ext.	CCBot	llms.txt
622	rajasthan.gov.in	Open	allowed	allowed	allowed	allowed	allowed	allowed	allowed	allowed	–
637	gst.gov.in	Unreadable	unknown	unknown	unknown	unknown	unknown	unknown	unknown	unknown	–
669	incometax.gov.in	Unreadable	unknown	unknown	unknown	unknown	unknown	unknown	unknown	unknown	–
681	uidai.gov.in	Open	allowed	allowed	allowed	allowed	allowed	allowed	allowed	allowed	–
715	epfindia.gov.in	Unreadable	unknown	unknown	unknown	unknown	unknown	unknown	unknown	unknown	–
720	eci.gov.in	No robots.txt	allowed	allowed	allowed	allowed	allowed	allowed	allowed	allowed	–
740	aajtak.in	Open	allowed	allowed	allowed	allowed	allowed	allowed	allowed	allowed	–
744	goodreturns.in	Open	allowed	allowed	allowed	allowed	allowed	allowed	allowed	allowed	–
746	indianrail.gov.in	Open	allowed	allowed	allowed	allowed	allowed	allowed	allowed	allowed	–
750	parivahan.gov.in	No robots.txt	allowed	allowed	allowed	allowed	allowed	allowed	allowed	allowed	–
769	screener.in	Open	allowed	allowed	allowed	allowed	allowed	allowed	allowed	allowed	–
811	digitallocker.gov.in	Unreadable	unknown	unknown	unknown	unknown	unknown	unknown	unknown	unknown	–
814	dord.gov.in	Unreadable	unknown	unknown	unknown	unknown	unknown	unknown	unknown	unknown	–
828	linkmake.in	Open	allowed	allowed	allowed	allowed	allowed	allowed	allowed	allowed	–
983	lnkd.in	Blocks some AI search	blocked	partial	blocked	blocked	partial	blocked	blocked	blocked	–
1000	educationportal3.in	No robots.txt	allowed	allowed	allowed	allowed	allowed	allowed	allowed	allowed	–
1539	registry.in	Open	allowed	allowed	allowed	allowed	allowed	allowed	allowed	allowed	–
2059	indiatoday.in	Open	allowed	allowed	allowed	allowed	allowed	allowed	allowed	allowed	–
2369	wcd.gov.in	Open	allowed	allowed	allowed	allowed	allowed	allowed	allowed	allowed	–
2417	admissions.nic.in	No robots.txt	allowed	allowed	allowed	allowed	allowed	allowed	allowed	allowed	–
2419	examinationservices.nic.in	No robots.txt	allowed	allowed	allowed	allowed	allowed	allowed	allowed	allowed	–
2872	airtel.in	Open	allowed	allowed	allowed	allowed	allowed	allowed	allowed	allowed	yes
3284	groww.in	Open	allowed	allowed	allowed	allowed	allowed	allowed	allowed	allowed	–
3432	kerala.gov.in	No robots.txt	allowed	allowed	allowed	allowed	allowed	allowed	allowed	allowed	–
3587	ctrls.in	No robots.txt	allowed	allowed	allowed	allowed	allowed	allowed	allowed	allowed	–
3887	bajajfinserv.in	Unreadable	unknown	unknown	unknown	unknown	unknown	unknown	unknown	unknown	–
3943	irctc.co.in	No robots.txt	allowed	allowed	allowed	allowed	allowed	allowed	allowed	allowed	–
4063	nobroker.in	Open	allowed	allowed	allowed	allowed	allowed	allowed	allowed	allowed	–
4162	mohfw.gov.in	Unreadable	unknown	unknown	unknown	unknown	unknown	unknown	unknown	unknown	–
4287	maharashtra.gov.in	Open	allowed	allowed	allowed	allowed	allowed	allowed	allowed	allowed	–
4339	jharkhand.gov.in	Open	allowed	allowed	allowed	allowed	allowed	allowed	allowed	allowed	–
4401	porter.in	Unreadable	unknown	unknown	unknown	unknown	unknown	unknown	unknown	unknown	–
4483	ernet.in	Open	allowed	allowed	allowed	allowed	allowed	allowed	allowed	allowed	–
4533	mxplayer.in	No robots.txt	allowed	allowed	allowed	allowed	allowed	allowed	allowed	allowed	–
4535	district.in	Open	allowed	allowed	allowed	allowed	allowed	allowed	allowed	allowed	–
4545	indiapost.gov.in	No robots.txt	allowed	allowed	allowed	allowed	allowed	allowed	allowed	allowed	–
4631	npci.org.in	Blocks some AI search	allowed	allowed	allowed	allowed	blocked	allowed	allowed	allowed	–
4853	nptel.ac.in	Open	allowed	allowed	allowed	allowed	allowed	allowed	allowed	allowed	–
4862	cashify.in	Open	allowed	allowed	allowed	allowed	allowed	allowed	allowed	allowed	yes
4892	digilocker.gov.in	Unreadable	unknown	unknown	unknown	unknown	unknown	unknown	unknown	unknown	–
5075	swayam.gov.in	Open	allowed	allowed	allowed	allowed	allowed	allowed	allowed	allowed	–
5129	railyatri.in	Open	allowed	allowed	allowed	allowed	allowed	allowed	allowed	allowed	–
5184	apollopharmacy.in	Open	allowed	allowed	allowed	allowed	allowed	allowed	allowed	allowed	yes
5258	scholarships.gov.in	No robots.txt	allowed	allowed	allowed	allowed	allowed	allowed	allowed	allowed	–
5294	samarth.edu.in	Open	allowed	allowed	allowed	allowed	allowed	allowed	allowed	allowed	–
5524	reliancedigital.in	Unreadable	unknown	unknown	unknown	unknown	unknown	unknown	unknown	unknown	–
5564	tanishq.co.in	Open	allowed	allowed	allowed	allowed	allowed	allowed	allowed	allowed	–
5651	shopsy.in	Open	allowed	allowed	allowed	allowed	allowed	allowed	allowed	allowed	–
5692	proteantech.in	No robots.txt	allowed	allowed	allowed	allowed	allowed	allowed	allowed	allowed	–
5701	nha.gov.in	Unreadable	unknown	unknown	unknown	unknown	unknown	unknown	unknown	unknown	–
5731	udyamregistration.gov.in	No robots.txt	allowed	allowed	allowed	allowed	allowed	allowed	allowed	allowed	–
5748	allahabadhighcourt.in	No robots.txt	allowed	allowed	allowed	allowed	allowed	allowed	allowed	allowed	–
5749	learncbse.in	Open	allowed	allowed	allowed	allowed	allowed	allowed	allowed	allowed	–
5756	pgportal.gov.in	No robots.txt	allowed	allowed	allowed	allowed	allowed	allowed	allowed	allowed	–
5775	haryana.gov.in	No robots.txt	allowed	allowed	allowed	allowed	allowed	allowed	allowed	allowed	–
5849	mybharat.gov.in	Open	allowed	allowed	allowed	allowed	allowed	allowed	allowed	allowed	–
5905	udiseplus.gov.in	No robots.txt	allowed	allowed	allowed	allowed	allowed	allowed	allowed	allowed	–
5933	meripehchaan.gov.in	No robots.txt	allowed	allowed	allowed	allowed	allowed	allowed	allowed	allowed	–
6002	cisf.gov.in	Open	allowed	allowed	allowed	allowed	allowed	allowed	allowed	allowed	–
6078	upbhulekh.gov.in	No robots.txt	allowed	allowed	allowed	allowed	allowed	allowed	allowed	allowed	–
6097	uod.ac.in	No robots.txt	allowed	allowed	allowed	allowed	allowed	allowed	allowed	allowed	–
6142	poshantracker.in	Unreadable	unknown	unknown	unknown	unknown	unknown	unknown	unknown	unknown	–
6201	tnpscexams.in	Unreadable	unknown	unknown	unknown	unknown	unknown	unknown	unknown	unknown	–
6203	up.nic.in	Open	allowed	allowed	allowed	allowed	allowed	allowed	allowed	allowed	–
6213	majhinaukri.in	Open	allowed	allowed	allowed	allowed	allowed	allowed	allowed	allowed	–
6235	igrsup.gov.in	No robots.txt	allowed	allowed	allowed	allowed	allowed	allowed	allowed	allowed	–
6249	ocbnyutpfwjs.in	Unreadable	unknown	unknown	unknown	unknown	unknown	unknown	unknown	unknown	–
6256	ncert.nic.in	Unreadable	unknown	unknown	unknown	unknown	unknown	unknown	unknown	unknown	–
6272	callbomberz.in	Open	allowed	allowed	allowed	allowed	allowed	allowed	allowed	allowed	–
6273	manakonline.in	Unreadable	unknown	unknown	unknown	unknown	unknown	unknown	unknown	unknown	–
6274	pmvidyalaxmi.co.in	Open	allowed	allowed	allowed	allowed	allowed	allowed	allowed	allowed	–
6289	lotterysambadresult.in	Open	allowed	allowed	allowed	allowed	allowed	allowed	allowed	allowed	–
6296	nrega.nic.in	No robots.txt	allowed	allowed	allowed	allowed	allowed	allowed	allowed	allowed	–
6378	qbrik.in	Open	allowed	allowed	allowed	allowed	allowed	allowed	allowed	allowed	–
6381	showpm.in	Open	allowed	allowed	allowed	allowed	allowed	allowed	allowed	allowed	–
6406	pib.gov.in	Unreadable	unknown	unknown	unknown	unknown	unknown	unknown	unknown	unknown	–
6555	cybercrime.gov.in	No robots.txt	allowed	allowed	allowed	allowed	allowed	allowed	allowed	allowed	–
6663	teletype.in	Open	allowed	allowed	allowed	allowed	allowed	allowed	allowed	allowed	–
6794	lpu.in	Open	allowed	allowed	allowed	allowed	allowed	allowed	allowed	allowed	yes
7040	ipowatch.in	Open	allowed	allowed	allowed	allowed	allowed	allowed	allowed	allowed	yes
7085	faymas.in	Open	allowed	allowed	allowed	allowed	allowed	allowed	allowed	allowed	yes
7206	shillongteergrounds.in	Open	allowed	allowed	allowed	allowed	allowed	allowed	allowed	allowed	–
7508	sweatco.in	Open	allowed	allowed	allowed	allowed	allowed	allowed	allowed	allowed	–
7546	angelone.in	Open	allowed	allowed	allowed	allowed	allowed	allowed	allowed	allowed	yes
7987	directverify.in	No robots.txt	allowed	allowed	allowed	allowed	allowed	allowed	allowed	allowed	–
8109	allevents.in	Open	allowed	allowed	allowed	partial	allowed	allowed	allowed	allowed	yes
8226	brainly.in	Blocks training only	blocked	allowed	allowed	allowed	allowed	allowed	allowed	allowed	–
8432	businesstoday.in	Open	allowed	allowed	allowed	allowed	allowed	allowed	allowed	allowed	–
8809	theprint.in	Blocks training only	blocked	allowed	allowed	allowed	allowed	allowed	allowed	blocked	–
8925	rbi.org.in	Unreadable	unknown	unknown	unknown	unknown	unknown	unknown	unknown	unknown	–
9415	indianvisaonline.gov.in	No robots.txt	allowed	allowed	allowed	allowed	allowed	allowed	allowed	allowed	–
9653	cbse.gov.in	Unreadable	unknown	unknown	unknown	unknown	unknown	unknown	unknown	unknown	–
9719	passportindia.gov.in	No robots.txt	allowed	allowed	allowed	allowed	allowed	allowed	allowed	allowed	–

Categories on .in (India)

Other 54 sites
Government 34 sites
Education 3 sites
Entertainment 1 sites
E-commerce 1 sites

FAQ

Do websites in India (.in) block AI crawlers?: As of 2026-10-02, 4.1% of the 74 websites in India (.in) whose robots.txt we could read block GPTBot, 1.4% block ClaudeBot and 1.4% block PerplexityBot (all 8,366 sites: 12%, 11%, 8.3%). 8 of them (10.8%) serve an llms.txt (all sites: 15.2%).
Which .in (India) websites block the most AI crawlers?: lnkd.in (26 of 29), npci.org.in (18 of 29), brainly.in (2 of 29), theprint.in (2 of 29). The full per-site status is on each site page.
How is this measured?: We fetch each site's robots.txt every day from one residential IP in Taiwan with an honest user agent, and apply the rules to 29 AI crawlers for the home page. 93 .in (India) sites are in the sample, 74 had a readable robots.txt (19 did not answer or returned an error page).

Other countries and suffixes

.com GPTBot 12.4%
.org GPTBot 11.3%
.net GPTBot 13.4%
.ru (Russia) GPTBot 5.1%
.de (Germany) GPTBot 23.6%
.jp (Japan) GPTBot 6%
.io GPTBot 4.3%
.edu GPTBot 3.7%
.gov GPTBot 4.6%
.br (Brazil) GPTBot 5.6%
.uk (United Kingdom) GPTBot 20%
.fr (France) GPTBot 31.8%
.cn (China) GPTBot 3.7%
.it (Italy) GPTBot 27.3%
.ai GPTBot 4.8%
.pl (Poland) GPTBot 26.4%
.au (Australia) GPTBot 27.8%
.tv GPTBot 6%
.nl (Netherlands) GPTBot 27.1%
.es (Spain) GPTBot 22.2%
.co GPTBot 9.1%
.ca (Canada) GPTBot 7.1%
.ch (Switzerland) GPTBot 16.7%
.ir (Iran) GPTBot 0%

Compare all suffixes