How do I block or allow Bytespider on my site?

Add "User-agent: Bytespider" followed by "Disallow: /" to https:// /robots.txt to block it, or "Allow: /" to allow it. Check the result with the free AI crawler checker at tools.yukai.uk/free/ai-crawler-checker.

AI Crawler Index · ByteDance · Bytespider

Which .org websites block Bytespider?

As of 2026-10-03, 40 of the 514 .org websites whose robots.txt we could read (7.8%) block Bytespider, ByteDance's AI training crawler, on the home page, compared with 12.3% of all sites; 1 more block it on some paths.

7.8%

of 514 .org websites block Bytespider

12.3%

of all sites

sites block it

All 40 .org websites that block Bytespider

#	Site	Policy	Crawlers blocked
275	unesco.org	Blocks training only	8 of 29
479	pbs.org	Blocks some AI search	4 of 29
604	themoviedb.org	Blocks some AI search	20 of 29
749	tjk.org	Blocks some AI search	28 of 29
1109	openssl.org	Blocks all AI search	29 of 29
1395	tmdb.org	Blocks some AI search	20 of 29
1434	aarp.org	Blocks training only	2 of 29
1743	unep.org	Blocks some AI search	11 of 29
1760	geeksforgeeks.org	Blocks training only	3 of 29
1964	codeberg.org	Blocks some AI search	12 of 29
2315	emojipedia.org	Blocks training only	7 of 29
2390	tvtropes.org	Blocks some AI search	11 of 29
2663	eurekalert.org	Blocks training only	1 of 29
2671	biorxiv.org	Blocks some AI search	8 of 29
3080	wto.org	Blocks some AI search	28 of 29
3131	gentoo.org	Blocks some AI search	20 of 29
3141	bookshop.org	Blocks training only	9 of 29
3684	csis.org	Blocks training only	1 of 29
3916	poetryfoundation.org	Blocks some AI search	11 of 29
3920	britishmuseum.org	Blocks some AI search	11 of 29
4260	albumoftheyear.org	Blocks training only	7 of 29
4715	nypl.org	Blocks training only	2 of 29
5173	fischipedia.org	Blocks training only	1 of 29
5193	sciencenews.org	Blocks some AI search	10 of 29
5434	nrdc.org	Blocks training only	3 of 29
5530	c-span.org	Blocks some AI search	8 of 29
5793	bangumi.org	Blocks some AI search	6 of 29
5928	wikicv.org	Blocks some AI search	28 of 29
5937	wbur.org	Blocks some AI search	13 of 29
6291	sgxnifty.org	Blocks training only	1 of 29
6516	healthychildren.org	Blocks training only	4 of 29
6690	pbskids.org	Blocks training only	8 of 29
7110	ebird.org	Blocks all AI search	29 of 29
7263	rsf.org	Blocks some AI search	3 of 29
7315	usgbc.org	Blocks some AI search	28 of 29
7474	f-droid.org	Blocks some AI search	11 of 29
8723	sae.org	Blocks training only	2 of 29
9189	project-syndicate.org	Blocks some AI search	10 of 29
9228	sil.org	Blocks some AI search	2 of 29
9901	adl.org	Blocks some AI search	9 of 29

Popular .org websites that allow Bytespider

Other AI crawlers and .org websites

Bytespider in other segments

Bytespider overview · All .org websites

FAQ

Which .org websites block Bytespider?: unesco.org, pbs.org, themoviedb.org, tjk.org, openssl.org, tmdb.org, aarp.org, unep.org, and 32 more (full list on this page).
How many .org websites block Bytespider?: As of 2026-10-03, 40 of the 514 .org websites whose robots.txt we could read (7.8%) block Bytespider, ByteDance's AI training crawler, on the home page, compared with 12.3% of all sites; 1 more block it on some paths.
How do I block or allow Bytespider on my site?: Add "User-agent: Bytespider" followed by "Disallow: /" to https://<your-site>/robots.txt to block it, or "Allow: /" to allow it. Check the result with the free AI crawler checker at tools.yukai.uk/free/ai-crawler-checker.