Which .io websites block the most AI crawlers?

huntress.io (29 of 29), lunalabs.io (29 of 29), quay.io (29 of 29), uakinogo.io (29 of 29), radar.io (29 of 29). The full per-site status is on each site page.

How is this measured?

We fetch each site's robots.txt every day from one residential IP in Taiwan with an honest user agent, and apply the rules to 29 AI crawlers for the home page. 200 .io sites are in the sample, 185 had a readable robots.txt (15 did not answer or returned an error page).

AI Crawler Index · domain suffix

Do .io websites block AI crawlers?

As of 2026-10-03, 4.3% of the 185 .io websites whose robots.txt we could read block GPTBot, 3.8% block ClaudeBot and 3.8% block PerplexityBot (all 8,357 sites: 12%, 11%, 8.3%). 62 of them (33.5%) serve an llms.txt (all sites: 15.2%).

4.3%

block GPTBot (all sites 12%)

3.8%

block ClaudeBot (all sites 11%)

33.5%

serve an llms.txt (all sites 15.2%)

185

of 200 sites with a readable robots.txt

Block rate by crawler: .io vs all sites

Crawler	Operator	.io	All sites	Difference
Bytespider	ByteDance	5.9%	12.3%	-6.4 pp
CCBot	Common Crawl	4.9%	12.8%	-7.9 pp
cohere-ai	Cohere	4.9%	9.1%	-4.2 pp
Meta-ExternalAgent	Meta	4.9%	9.5%	-4.6 pp
GPTBot	OpenAI	4.3%	12%	-7.7 pp
Amazonbot	Amazon	3.8%	9.2%	-5.4 pp
Applebot-Extended	Apple	3.8%	9.6%	-5.8 pp
ClaudeBot	Anthropic	3.8%	11%	-7.2 pp
Google-Extended	Google	3.8%	9.5%	-5.7 pp
PerplexityBot	Perplexity	3.8%	8.3%	-4.5 pp
Amzn-SearchBot	Amazon	3.2%	4.1%	-0.9 pp
Amzn-User	Amazon	3.2%	3.9%	-0.7 pp
Applebot	Apple	3.2%	3.1%	+0.1 pp
ChatGPT-User	OpenAI	3.2%	7.5%	-4.3 pp
Claude-SearchBot	Anthropic	3.2%	5.4%	-2.2 pp
Claude-User	Anthropic	3.2%	5.5%	-2.3 pp
DuckAssistBot	DuckDuckGo	3.2%	5.8%	-2.6 pp
Google-Agent	Google	3.2%	3.2%	0 pp
Google-CloudVertexBot	Google	3.2%	5.2%	-2 pp
Google-GeminiNotebook	Google	3.2%	3%	+0.2 pp
meta-externalads	Meta	3.2%	2.8%	+0.4 pp
meta-externalfetcher	Meta	3.2%	6%	-2.8 pp
meta-webindexer	Meta	3.2%	4.6%	-1.4 pp
MistralAI-Index	Mistral	3.2%	3.3%	-0.1 pp
MistralAI-Training	Mistral	3.2%	3.3%	-0.1 pp
MistralAI-User	Mistral	3.2%	5.4%	-2.2 pp
OAI-AdsBot	OpenAI	3.2%	3%	+0.2 pp
OAI-SearchBot	OpenAI	3.2%	5.7%	-2.5 pp
Perplexity-User	Perplexity	3.2%	5.7%	-2.5 pp

Who blocks what, by name: GPTBot, ClaudeBot, PerplexityBot, Google-Extended, CCBot, Bytespider, Applebot-Extended, OAI-SearchBot.

Policy mix

Policy	Sites	Share
Open	134	67%
No robots.txt	39	19.5%
Unreadable	15	7.5%
Blocks all AI search	6	3%
Blocks training only	5	2.5%
Blocks some AI search	1	0.5%

Sites with an llms.txt (62)

Top 100 of 200 .io sites

✕ blocked · ◐ partially blocked (some paths) · ✓ allowed · ? robots.txt unreadable. "#" is the position in our popularity-ordered sample.

#	Site	Policy	GPTBot	OAI-Search	ChatGPT-User	ClaudeBot	Claude-Search	Perplexity	Google-Ext.	CCBot	llms.txt
41	sentry.io	Open	allowed	allowed	allowed	allowed	allowed	allowed	allowed	allowed	yes
290	intercom.io	Open	allowed	allowed	allowed	allowed	allowed	allowed	allowed	allowed	yes
308	wpguardian.io	Open	allowed	allowed	allowed	allowed	allowed	allowed	allowed	allowed	–
321	oxylabs.io	Open	allowed	allowed	allowed	allowed	allowed	allowed	allowed	allowed	yes
325	itch.io	Open	allowed	allowed	allowed	allowed	allowed	allowed	allowed	allowed	–
385	pendo.io	Open	allowed	allowed	allowed	allowed	allowed	allowed	allowed	allowed	yes
406	agora.io	Open	allowed	allowed	allowed	allowed	allowed	allowed	allowed	allowed	yes
591	0xrpc.io	No robots.txt	allowed	allowed	allowed	allowed	allowed	allowed	allowed	allowed	–
663	cardgames.io	Open	allowed	allowed	allowed	allowed	allowed	allowed	allowed	allowed	–
712	ssstik.io	Open	allowed	allowed	allowed	allowed	allowed	allowed	allowed	allowed	–
897	astrumu.io	Open	allowed	allowed	allowed	allowed	allowed	allowed	allowed	allowed	–
1004	privacymanager.io	Open	allowed	allowed	allowed	allowed	allowed	allowed	allowed	allowed	yes
1070	nic.io	No robots.txt	allowed	allowed	allowed	allowed	allowed	allowed	allowed	allowed	–
1083	shodan.io	Open	allowed	allowed	allowed	allowed	allowed	allowed	allowed	allowed	yes
1086	metamask.io	Open	allowed	allowed	allowed	allowed	allowed	allowed	allowed	allowed	yes
1143	customer.io	Open	allowed	allowed	allowed	allowed	allowed	allowed	allowed	allowed	yes
1237	trustindex.io	Open	allowed	allowed	allowed	allowed	allowed	allowed	allowed	allowed	–
1245	split.io	Open	allowed	allowed	allowed	allowed	allowed	allowed	allowed	allowed	–
1274	home-assistant.io	Open	allowed	allowed	allowed	allowed	allowed	allowed	allowed	allowed	–
1296	snapcraft.io	Open	allowed	allowed	allowed	allowed	allowed	allowed	allowed	allowed	yes
1308	sslip.io	No robots.txt	allowed	allowed	allowed	allowed	allowed	allowed	allowed	allowed	–
1379	aditude.io	Open	allowed	allowed	allowed	allowed	allowed	allowed	allowed	allowed	–
1381	nexx360.io	Open	allowed	allowed	allowed	allowed	allowed	allowed	allowed	allowed	–
1412	mediago.io	No robots.txt	allowed	allowed	allowed	allowed	allowed	allowed	allowed	allowed	–
1429	redis.io	Open	allowed	allowed	allowed	allowed	allowed	allowed	allowed	allowed	yes
1435	huntress.io	Blocks all AI search	blocked	blocked	blocked	blocked	blocked	blocked	blocked	blocked	–
1467	cloudx.io	Open	allowed	allowed	allowed	allowed	allowed	allowed	allowed	allowed	–
1600	avs.io	Open	allowed	allowed	allowed	allowed	allowed	allowed	allowed	allowed	–
1652	merkle.io	No robots.txt	allowed	allowed	allowed	allowed	allowed	allowed	allowed	allowed	–
1692	guard.io	Open	allowed	allowed	allowed	allowed	allowed	allowed	allowed	allowed	yes
1720	nip.io	No robots.txt	allowed	allowed	allowed	allowed	allowed	allowed	allowed	allowed	–
1801	azurecontainerapps.io	No robots.txt	allowed	allowed	allowed	allowed	allowed	allowed	allowed	allowed	–
1806	pantheonsite.io	Unreadable	unknown	unknown	unknown	unknown	unknown	unknown	unknown	unknown	–
1845	gitbook.io	Unreadable	unknown	unknown	unknown	unknown	unknown	unknown	unknown	unknown	–
1887	ably.io	Open	allowed	allowed	allowed	allowed	allowed	allowed	allowed	allowed	yes
1896	omni-dex.io	Open	allowed	allowed	allowed	allowed	allowed	allowed	allowed	allowed	–
2020	elevenlabs.io	Open	allowed	allowed	allowed	allowed	allowed	allowed	allowed	allowed	yes
2039	island.io	Open	allowed	allowed	allowed	allowed	allowed	allowed	allowed	allowed	–
2062	piano.io	Open	allowed	allowed	allowed	allowed	allowed	allowed	allowed	allowed	yes
2094	connectad.io	Open	allowed	allowed	allowed	allowed	allowed	allowed	allowed	allowed	–
2107	plausible.io	Open	allowed	allowed	allowed	allowed	allowed	allowed	allowed	allowed	–
2110	vimla.io	Unreadable	unknown	unknown	unknown	unknown	unknown	unknown	unknown	unknown	–
2121	complianz.io	Open	allowed	allowed	allowed	allowed	allowed	allowed	allowed	allowed	yes
2138	netbird.io	Open	allowed	allowed	allowed	allowed	allowed	allowed	allowed	allowed	–
2181	codepen.io	Unreadable	unknown	unknown	unknown	unknown	unknown	unknown	unknown	unknown	–
2233	missena.io	No robots.txt	allowed	allowed	allowed	allowed	allowed	allowed	allowed	allowed	–
2317	jumpjump.io	Open	allowed	allowed	allowed	allowed	allowed	allowed	allowed	allowed	–
2322	growthbook.io	Open	allowed	allowed	allowed	allowed	allowed	allowed	allowed	allowed	yes
2341	sanity.io	Open	allowed	allowed	allowed	allowed	allowed	allowed	allowed	allowed	yes
2351	termly.io	Open	allowed	allowed	allowed	allowed	allowed	allowed	allowed	allowed	yes
2357	bitdrift.io	Open	allowed	allowed	allowed	allowed	allowed	allowed	allowed	allowed	–
2371	kandji.io	Open	allowed	allowed	allowed	allowed	allowed	allowed	allowed	allowed	–
2384	advolve.io	No robots.txt	allowed	allowed	allowed	allowed	allowed	allowed	allowed	allowed	–
2422	greenhouse.io	Open	allowed	allowed	allowed	allowed	allowed	allowed	allowed	allowed	yes
2473	gohugo.io	No robots.txt	allowed	allowed	allowed	allowed	allowed	allowed	allowed	allowed	–
2494	opensea.io	Open	allowed	allowed	allowed	allowed	allowed	allowed	allowed	allowed	yes
2749	fpjs.io	Open	allowed	allowed	allowed	allowed	allowed	allowed	allowed	allowed	yes
2769	instana.io	No robots.txt	allowed	allowed	allowed	allowed	allowed	allowed	allowed	allowed	–
2800	ghcr.io	No robots.txt	allowed	allowed	allowed	allowed	allowed	allowed	allowed	allowed	–
2857	kubernetes.io	Open	allowed	allowed	allowed	allowed	allowed	allowed	allowed	allowed	–
3019	ussp.io	Open	allowed	allowed	allowed	allowed	allowed	allowed	allowed	allowed	–
3028	gofile.io	Open	allowed	allowed	allowed	allowed	allowed	allowed	allowed	allowed	–
3031	ves.io	Unreadable	unknown	unknown	unknown	unknown	unknown	unknown	unknown	unknown	–
3133	nownodes.io	Open	allowed	allowed	allowed	allowed	allowed	allowed	allowed	allowed	yes
3136	infura.io	Open	allowed	allowed	allowed	allowed	allowed	allowed	allowed	allowed	–
3144	etherscan.io	Open	allowed	allowed	allowed	allowed	allowed	allowed	allowed	allowed	yes
3165	blastapi.io	No robots.txt	allowed	allowed	allowed	allowed	allowed	allowed	allowed	allowed	–
3272	zeta-ai.io	Open	allowed	allowed	allowed	allowed	allowed	allowed	allowed	allowed	–
3298	mevblocker.io	No robots.txt	allowed	allowed	allowed	allowed	allowed	allowed	allowed	allowed	–
3311	bidmatic.io	Open	allowed	allowed	allowed	allowed	allowed	allowed	allowed	allowed	–
3416	5ch.io	Open	allowed	allowed	allowed	allowed	allowed	allowed	allowed	allowed	–
3419	osf.io	Blocks some AI search	blocked	allowed	allowed	allowed	allowed	blocked	allowed	allowed	–
3470	n8n.io	Open	allowed	allowed	allowed	allowed	allowed	allowed	allowed	allowed	yes
3673	lunalabs.io	Blocks all AI search	blocked	blocked	blocked	blocked	blocked	blocked	blocked	blocked	–
3687	1rpc.io	No robots.txt	allowed	allowed	allowed	allowed	allowed	allowed	allowed	allowed	–
3767	airbridge.io	Blocks training only	partial	partial	partial	partial	allowed	partial	partial	blocked	yes
3844	cloudez.io	Open	allowed	allowed	allowed	allowed	allowed	allowed	allowed	allowed	–
3853	smooch.io	No robots.txt	allowed	allowed	allowed	allowed	allowed	allowed	allowed	allowed	–
3869	karte.io	Unreadable	unknown	unknown	unknown	unknown	unknown	unknown	unknown	unknown	–
3902	transcend.io	Open	allowed	allowed	allowed	allowed	allowed	allowed	allowed	allowed	yes
3964	meduza.io	Open	allowed	allowed	allowed	allowed	allowed	allowed	allowed	allowed	–
3994	exodus.io	Open	allowed	allowed	allowed	allowed	allowed	allowed	allowed	allowed	yes
4028	jfrog.io	No robots.txt	allowed	allowed	allowed	allowed	allowed	allowed	allowed	allowed	–
4067	datagrail.io	Open	allowed	allowed	allowed	allowed	allowed	allowed	allowed	allowed	–
4071	pinklion.io	Open	allowed	allowed	allowed	allowed	allowed	allowed	allowed	allowed	–
4370	smile.io	Open	allowed	allowed	allowed	allowed	allowed	allowed	allowed	allowed	yes
4388	quay.io	Blocks all AI search	blocked	blocked	blocked	blocked	blocked	blocked	blocked	blocked	–
4442	apollo.io	Open	allowed	allowed	allowed	allowed	allowed	allowed	allowed	allowed	yes
4452	gwars.io	Open	allowed	allowed	allowed	allowed	allowed	allowed	allowed	allowed	–
4525	notvpn.io	No robots.txt	allowed	allowed	allowed	allowed	allowed	allowed	allowed	allowed	–
4583	gravitational.io	No robots.txt	allowed	allowed	allowed	allowed	allowed	allowed	allowed	allowed	–
4622	trezor.io	Open	allowed	allowed	allowed	allowed	allowed	allowed	allowed	allowed	yes
4691	digitalaudience.io	Open	allowed	allowed	allowed	allowed	allowed	allowed	allowed	allowed	–
4724	coda.io	Open	allowed	allowed	allowed	allowed	allowed	allowed	allowed	allowed	–
4785	prismic.io	Open	allowed	allowed	allowed	allowed	allowed	allowed	allowed	allowed	yes
4854	envoyer.io	Open	allowed	allowed	allowed	allowed	allowed	allowed	allowed	allowed	–
5034	kaliscan.io	Open	allowed	allowed	allowed	allowed	allowed	allowed	allowed	allowed	–
5135	receptivity.io	Open	allowed	allowed	allowed	allowed	allowed	allowed	allowed	allowed	yes
5363	kayzen.io	Open	allowed	allowed	allowed	allowed	allowed	allowed	allowed	allowed	–
5367	jenkins.io	No robots.txt	allowed	allowed	allowed	allowed	allowed	allowed	allowed	allowed	–

Sorted by popularity; the rest are on the A-Z lists.

Categories on .io

Other 191 sites
Entertainment 4 sites
Tech 3 sites
Finance 1 sites
News & media 1 sites

FAQ

Do .io websites block AI crawlers?: As of 2026-10-03, 4.3% of the 185 .io websites whose robots.txt we could read block GPTBot, 3.8% block ClaudeBot and 3.8% block PerplexityBot (all 8,357 sites: 12%, 11%, 8.3%). 62 of them (33.5%) serve an llms.txt (all sites: 15.2%).
Which .io websites block the most AI crawlers?: huntress.io (29 of 29), lunalabs.io (29 of 29), quay.io (29 of 29), uakinogo.io (29 of 29), radar.io (29 of 29). The full per-site status is on each site page.
How is this measured?: We fetch each site's robots.txt every day from one residential IP in Taiwan with an honest user agent, and apply the rules to 29 AI crawlers for the home page. 200 .io sites are in the sample, 185 had a readable robots.txt (15 did not answer or returned an error page).

Other countries and suffixes

.com GPTBot 12.4%
.org GPTBot 11.1%
.net GPTBot 13.6%
.ru (Russia) GPTBot 5.1%
.de (Germany) GPTBot 23.6%
.jp (Japan) GPTBot 6%
.edu GPTBot 3.7%
.gov GPTBot 4.6%
.br (Brazil) GPTBot 6.7%
.uk (United Kingdom) GPTBot 20%
.fr (France) GPTBot 31.8%
.in (India) GPTBot 4.1%
.cn (China) GPTBot 3.8%
.it (Italy) GPTBot 27.3%
.ai GPTBot 4.9%
.pl (Poland) GPTBot 26.4%
.au (Australia) GPTBot 27%
.tv GPTBot 6%
.nl (Netherlands) GPTBot 27.1%
.es (Spain) GPTBot 25%
.co GPTBot 8.9%
.ca (Canada) GPTBot 7.1%
.ch (Switzerland) GPTBot 16.7%
.ir (Iran) GPTBot 0%

Compare all suffixes