Bytespider is ByteDance's AI training crawler: it collects pages to train AI models. It is the #2 most blocked of the 29 AI crawlers in the index.

How do I block Bytespider in robots.txt?

Add these lines to https:// /robots.txt: "User-agent: Bytespider" followed by "Disallow: /". To allow it, use "Allow: /" instead. Well-behaved crawlers read robots.txt before fetching pages; firewall rules are needed to stop crawlers that ignore it.

How do I check whether my site blocks Bytespider?

Use the free AI crawler checker at tools.yukai.uk/free/ai-crawler-checker: it shows the status of Bytespider and 28 other AI crawlers with the exact robots.txt rule.

AI crawler · ByteDance · Training

Which websites block Bytespider?

As of 2026-10-01, 151 of the 853 popular websites whose robots.txt we could read (17.7%) block Bytespider, ByteDance's AI training crawler, on the home page; 21 more block it on some paths. Among the top 100 sites: 30.1%.

17.7%

of 853 readable sites block Bytespider

30.1%

of the top 100

most blocked of 29 AI crawlers

681

sites allow it on the home page

Blocked by category

Category	Sites read	Block Bytespider
News & media	126	52.4%
Social	48	27.1%
Reference	43	20.9%
E-commerce	58	15.5%
Entertainment	68	13.2%
Other	260	11.5%
AI	9	11.1%
Travel	13	7.7%
Tech	154	7.1%
Government	33	6.1%
Education	20	0%
Finance	16	0%
Telecom	5	0%

Popular sites that block Bytespider (151)

facebook.com Social
instagram.com Social
twitter.com Social
linkedin.com Social
amazon.com E-commerce
github.com Tech
netflix.com Entertainment
x.com Social
pinterest.com Social
tiktok.com Social
yahoo.com Tech
chatgpt.com AI
vimeo.com Entertainment
reddit.com Social
amazonvideo.com Entertainment
snapchat.com Social
unity3d.com Tech
nytimes.com News & media
flickr.com Other
medium.com News & media
theguardian.com News & media
ebay.com E-commerce
t-online.de News & media
forbes.com News & media
bbc.com News & media
canva.com Tech
meraki.com Tech
imdb.com Reference
launchpad.net Other
reuters.com News & media
nature.com Reference
weibo.com Social
weather.com News & media
hostinger.com Tech
ea.com Entertainment
trustpilot.com Other
bloomberg.com News & media
wsj.com News & media
espn.com News & media
unsplash.com Tech
temu.com E-commerce
businessinsider.com News & media
aol.com Tech
pixabay.com Tech
cnbc.com News & media
quora.com Reference
behance.net Social
ft.com News & media
wired.com News & media
uol.com.br News & media
usatoday.com News & media
tripadvisor.com Travel
threads.com Social
telegraph.co.uk News & media
unesco.org Government
nintendo.com Entertainment
techcrunch.com News & media
sagepub.com Reference
primevideo.com Entertainment
elpais.com News & media
dailymail.com News & media
lemonde.fr News & media
fb.com Other
cnet.com News & media
bandcamp.com E-commerce
bild.de News & media
ted.com Entertainment
welt.de News & media
people.com News & media
academia.edu Reference
theverge.com News & media
pexels.com Tech
figma.com Tech
nbcnews.com News & media
latimes.com News & media
onet.pl News & media
repubblica.it News & media
corriere.it News & media
investopedia.com News & media
flashscore.com News & media
healthline.com Reference
theatlantic.com News & media
letterboxd.com Reference
pbs.org News & media
skroutz.gr E-commerce
as.com News & media
bfmtv.com News & media
huffpost.com News & media
cookpad.com Entertainment
aftonbladet.se News & media
nypost.com News & media
economist.com News & media
theconversation.com News & media
ad.nl News & media
nu.nl News & media
elconfidencial.com News & media
vg.no News & media
indiamart.com E-commerce
spiegel.de News & media
leboncoin.fr E-commerce
dcinside.com Social
pandora.com Entertainment
hln.be News & media
emag.ro E-commerce
abc.net.au News & media
livescore.com News & media
themoviedb.org Reference
ladepeche.fr News & media
mydramalist.com Reference
actu.fr News & media
huffingtonpost.com News & media
dagbladet.no News & media
ilmeteo.it News & media
sapo.pt News & media
dw.com News & media
iltalehti.fi News & media
aljazeera.com News & media
tjk.org Other
buzzfeed.com Other
aka.ms Other
meteociel.fr News & media
publi24.ro Other
sportfilm800.com News & media
twkan.com Other
mobcup.fm Other
rocketmoney.dev Other
qcloud.com Other
newsweek.com News & media
newyorker.com Other
1c.ru Other
blackhub.team Other
arstechnica.com News & media
zdnet.com News & media
adrta.com Other
marketwatch.com E-commerce
merkur.de Other
thelancet.com Other
wikihow.com Other
oreilly.com Other
hola.com Other
jamanetwork.com Other
fastcompany.com Other
lnkd.in Other
meb.gov.tr Government
sogou.com Other
ign.com Other
gizmodo.com Other
salesforceliveagent.com Other
timeanddate.com Other
sedoparking.com Other
washingtonpost.com News & media

Trend

The daily series for Bytespider starts on 2026-10-01; a trend appears here as days accumulate.

Latest changes

dailymotion.com: blocked → allowed
figma.com: allowed → blocked

Block or allow Bytespider

User-agent: Bytespider
Disallow: /

Replace Disallow: / with Allow: / to let it in. A site that blocks training crawlers but wants to be cited in AI answers usually keeps the search and user-triggered crawlers allowed.

FAQ

How many websites block Bytespider?: As of 2026-10-01, 151 of the 853 popular websites whose robots.txt we could read (17.7%) block Bytespider, ByteDance's AI training crawler, on the home page; 21 more block it on some paths. Among the top 100 sites: 30.1%.
What is Bytespider?: Bytespider is ByteDance's AI training crawler: it collects pages to train AI models. It is the #2 most blocked of the 29 AI crawlers in the index.
How do I block Bytespider in robots.txt?: Add these lines to https://<your-site>/robots.txt: "User-agent: Bytespider" followed by "Disallow: /". To allow it, use "Allow: /" instead. Well-behaved crawlers read robots.txt before fetching pages; firewall rules are needed to stop crawlers that ignore it.
How do I check whether my site blocks Bytespider?: Use the free AI crawler checker at tools.yukai.uk/free/ai-crawler-checker: it shows the status of Bytespider and 28 other AI crawlers with the exact robots.txt rule.