CCBot is Common Crawl's AI training crawler: it collects pages to train AI models. It is the #1 most blocked of the 29 AI crawlers in the index.

How do I block CCBot in robots.txt?

Add these lines to https:// /robots.txt: "User-agent: CCBot" followed by "Disallow: /". To allow it, use "Allow: /" instead. Well-behaved crawlers read robots.txt before fetching pages; firewall rules are needed to stop crawlers that ignore it.

How do I check whether my site blocks CCBot?

Use the free AI crawler checker at tools.yukai.uk/free/ai-crawler-checker: it shows the status of CCBot and 28 other AI crawlers with the exact robots.txt rule.

AI crawler · Common Crawl · Training

Which websites block CCBot?

As of 2026-10-01, 166 of the 853 popular websites whose robots.txt we could read (19.5%) block CCBot, Common Crawl's AI training crawler, on the home page; 26 more block it on some paths. Among the top 100 sites: 26.5%.

19.5%

of 853 readable sites block CCBot

26.5%

of the top 100

most blocked of 29 AI crawlers

661

sites allow it on the home page

Blocked by category

Category	Sites read	Block CCBot
News & media	126	58.7%
Social	48	29.2%
Reference	43	27.9%
Entertainment	68	14.7%
E-commerce	58	13.8%
Other	260	13.5%
AI	9	11.1%
Travel	13	7.7%
Government	33	6.1%
Tech	154	5.8%
Education	20	0%
Finance	16	0%
Telecom	5	0%

Popular sites that block CCBot (166)

facebook.com Social
instagram.com Social
twitter.com Social
linkedin.com Social
amazon.com E-commerce
netflix.com Entertainment
x.com Social
pinterest.com Social
tiktok.com Social
yahoo.com Tech
chatgpt.com AI
vimeo.com Entertainment
reddit.com Social
amazonvideo.com Entertainment
snapchat.com Social
nytimes.com News & media
flickr.com Other
theguardian.com News & media
ebay.com E-commerce
t-online.de News & media
forbes.com News & media
bbc.com News & media
canva.com Tech
who.int Reference
imdb.com Reference
launchpad.net Other
reuters.com News & media
nature.com Reference
weibo.com Social
weather.com News & media
hostinger.com Tech
trustpilot.com Other
bloomberg.com News & media
wsj.com News & media
calendly.com Tech
espn.com News & media
globo.com News & media
businessinsider.com News & media
aol.com Tech
pixabay.com Tech
goodreads.com Reference
cnbc.com News & media
behance.net Social
eventbrite.com Other
ft.com News & media
wired.com News & media
uol.com.br News & media
usatoday.com News & media
tripadvisor.com Travel
threads.com Social
telegraph.co.uk News & media
unesco.org Government
nintendo.com Entertainment
techcrunch.com News & media
sagepub.com Reference
primevideo.com Entertainment
elpais.com News & media
dailymail.com News & media
lemonde.fr News & media
fb.com Other
cnet.com News & media
bandcamp.com E-commerce
wp.pl News & media
bild.de News & media
apnews.com News & media
ted.com Entertainment
welt.de News & media
people.com News & media
academia.edu Reference
theverge.com News & media
pexels.com Tech
figma.com Tech
hbr.org News & media
nbcnews.com News & media
repubblica.it News & media
webmd.com Reference
corriere.it News & media
investopedia.com News & media
pixiv.net Entertainment
elmundo.es News & media
flightradar24.com Entertainment
flashscore.com News & media
healthline.com Reference
theatlantic.com News & media
letterboxd.com Reference
themeforest.net E-commerce
pbs.org News & media
n-tv.de News & media
skroutz.gr E-commerce
archiveofourown.org Social
as.com News & media
bfmtv.com News & media
huffpost.com News & media
marca.com News & media
skysports.com News & media
cookpad.com Entertainment
aftonbladet.se News & media
gsmarena.com News & media
nypost.com News & media
economist.com News & media
theconversation.com News & media
ad.nl News & media
nu.nl News & media
elconfidencial.com News & media
vg.no News & media
spiegel.de News & media
dcinside.com Social
rightmove.co.uk E-commerce
pandora.com Entertainment
gazzetta.it News & media
hln.be News & media
emag.ro E-commerce
abc.net.au News & media
themoviedb.org Reference
fortune.com News & media
ladepeche.fr News & media
mydramalist.com Reference
actu.fr News & media
jstor.org Reference
nikkansports.com News & media
huffingtonpost.com News & media
dagbladet.no News & media
ilmeteo.it News & media
rustdesk.com Tech
sapo.pt News & media
dw.com News & media
iltalehti.fi News & media
ekstrabladet.dk News & media
programme-tv.net Other
bt.dk Other
tjk.org Other
buzzfeed.com Other
aka.ms Other
sportfilm800.com News & media
twkan.com Other
shutterstock.com Other
rocketmoney.dev Other
qcloud.com Other
newsweek.com News & media
newyorker.com Other
blackhub.team Other
arstechnica.com News & media
vinted.fr Other
zdnet.com News & media
adrta.com Other
marketwatch.com E-commerce
scientificamerican.com Other
ixl.com Other
merkur.de Other
sciencemag.org Other
thelancet.com Other
wikihow.com Other
oreilly.com Other
hola.com Other
euronews.com News & media
jamanetwork.com Other
fastcompany.com Other
lnkd.in Other
meb.gov.tr Government
sogou.com Other
science.org Other
ign.com Other
gizmodo.com Other
salesforceliveagent.com Other
sedoparking.com Other
washingtonpost.com News & media

Trend

The daily series for CCBot starts on 2026-10-01; a trend appears here as days accumulate.

Latest changes

dailymotion.com: blocked → allowed
eventbrite.com: allowed → blocked
eff.org: allowed → partial

Block or allow CCBot

User-agent: CCBot
Disallow: /

Replace Disallow: / with Allow: / to let it in. A site that blocks training crawlers but wants to be cited in AI answers usually keeps the search and user-triggered crawlers allowed.

FAQ

How many websites block CCBot?: As of 2026-10-01, 166 of the 853 popular websites whose robots.txt we could read (19.5%) block CCBot, Common Crawl's AI training crawler, on the home page; 26 more block it on some paths. Among the top 100 sites: 26.5%.
What is CCBot?: CCBot is Common Crawl's AI training crawler: it collects pages to train AI models. It is the #1 most blocked of the 29 AI crawlers in the index.
How do I block CCBot in robots.txt?: Add these lines to https://<your-site>/robots.txt: "User-agent: CCBot" followed by "Disallow: /". To allow it, use "Allow: /" instead. Well-behaved crawlers read robots.txt before fetching pages; firewall rules are needed to stop crawlers that ignore it.
How do I check whether my site blocks CCBot?: Use the free AI crawler checker at tools.yukai.uk/free/ai-crawler-checker: it shows the status of CCBot and 28 other AI crawlers with the exact robots.txt rule.