AI crawler · Meta · Training
Which websites block Meta-ExternalAgent?
As of 2026-11-16, 1,134 of the 8,313 popular websites whose robots.txt we could read (13.6%) block Meta-ExternalAgent, Meta's AI training crawler, on the home page; 133 more block it on some paths. Among the top 100 sites: 25.3%.
13.6%
of 8,313 readable sites block Meta-ExternalAgent
25.3%
of the top 100
#5
most blocked of 29 AI crawlers
7,046
sites allow it on the home page
Blocked by category
| Category | Sites read | Block Meta-ExternalAgent |
|---|---|---|
| Social | 51 | 33.3% |
| News & media | 392 | 30.6% |
| Entertainment | 332 | 19% |
| Reference | 43 | 18.6% |
| E-commerce | 116 | 14.7% |
| Other | 6651 | 13.3% |
| Travel | 13 | 7.7% |
| Finance | 102 | 6.9% |
| Education | 232 | 3.9% |
| Government | 203 | 2.5% |
| Tech | 162 | 1.2% |
| AI | 10 | 0% |
| Telecom | 6 | 0% |
Popular sites that block Meta-ExternalAgent (1,134)
- facebook.com Social
- instagram.com Social
- twitter.com Social
- linkedin.com Social
- amazon.com E-commerce
- netflix.com Entertainment
- x.com Social
- pinterest.com Social
- tiktok.com Social
- reddit.com Social
- amazonvideo.com Entertainment
- snapchat.com Social
- tumblr.com Social
- nytimes.com News & media
- flickr.com Other
- medium.com News & media
- theguardian.com News & media
- ebay.com E-commerce
- t-online.de News & media
- forbes.com News & media
- bbc.com News & media
- canva.com Tech
- imdb.com Reference
- launchpad.net Other
- reuters.com News & media
- nature.com Reference
- weibo.com Social
- name.com Other
- ea.com Entertainment
- washingtonpost.com News & media
- bloomberg.com News & media
- wsj.com News & media
- cnbc.com News & media
- behance.net Social
- indiatimes.com News & media
- ft.com News & media
- wired.com News & media
- uol.com.br News & media
- tripadvisor.com Travel
- usatoday.com News & media
- threads.com Social
- telegraph.co.uk News & media
- nintendo.com Entertainment
- sagepub.com Reference
- primevideo.com Entertainment
- dailymail.com News & media
- kick.com Entertainment
- fb.com Other
- cnet.com News & media
- bandcamp.com E-commerce
- zillow.com E-commerce
- ted.com Entertainment
- people.com News & media
- academia.edu Reference
- theverge.com News & media
- hbr.org News & media
- nbcnews.com News & media
- onet.pl News & media
- corriere.it News & media
- investopedia.com News & media
- pixiv.net Entertainment
- elmundo.es News & media
- healthline.com Reference
- theatlantic.com News & media
- letterboxd.com Reference
- n-tv.de News & media
- ouest-france.fr News & media
- huffpost.com News & media
- bfmtv.com News & media
- marca.com News & media
- aftonbladet.se News & media
- cookpad.com Entertainment
- nypost.com News & media
- theconversation.com News & media
- ad.nl News & media
- nu.nl News & media
- elconfidencial.com News & media
- vg.no News & media
- dcinside.com Social
- coursera.org Tech
- pandora.com Entertainment
- gazzetta.it News & media
- hln.be News & media
- abc.net.au News & media
- litnet.com Other
- themoviedb.org Reference
- mydramalist.com Reference
- spiegel.de News & media
- actu.fr News & media
- sapo.pt News & media
- huffingtonpost.com News & media
- dagbladet.no News & media
- syosetu.com Social
- ilmeteo.it News & media
- dw.com News & media
- iltalehti.fi News & media
- cardmarket.com E-commerce
- ekstrabladet.dk News & media
- 1mg.com Other
- dogdrip.net Social
- notion.com Other
- tjk.org Other
- aka.ms Other
- myreadingmanga.info Entertainment
- buzzfeed.com Other
- theqoo.net Other
- mamastar.jp Other
- twkan.com Other
- sportfilm800.com News & media
- shutterstock.com Other
- qcloud.com Other
- newyorker.com Other
- merkur.de Other
- arstechnica.com News & media
- zdnet.com News & media
- rocketmoney.dev Other
- adrta.com Other
- marketwatch.com E-commerce
- liquipedia.net Other
- sciencemag.org Other
- thelancet.com Other
- wikihow.com Other
- oreilly.com Other
- salthasher.com Other
- jamanetwork.com Other
- blackhub.team Other
- nontondrama.my Other
- lnkd.in Other
- strava.com Other
- vidoyu.space Other
- jhu.edu Education
- sedoparking.com Other
- sogou.com Other
- science.org Other
- bollyflix.free Other
- ign.com Other
- salesforceliveagent.com Other
- gizmodo.com Other
- adsco.re Other
- videeyss.shop E-commerce
- cukrarnauandela.cz Other
- moviezwap.taxi Entertainment
- filmyzilla63.com Entertainment
- filmyzilla56.com Entertainment
- filmyzilla62.com Entertainment
- timeanddate.com Other
- mashable.com Other
- thesun.co.uk Other
- fliphtml5.com Other
- proofpoint.com Other
- sap.com Other
- openssl.org Other
- medicalnewstoday.com News & media
- variety.com Other
- mirror.co.uk Other
- constellix.com Other
- techradar.com Other
- sendbird.com Other
- kasserver.com Other
- vox.com Other
- instacart.com Other
- congress.gov Government
- co.com Other
- iheart.com Other
- guardian.co.uk Other
- tappx.com Other
- rollingstone.com Other
- redditmedia.com Other
- slate.com Other
- tidal.com Other
- hollywoodreporter.com Other
- news.com.au News & media
- entrepreneur.com Other
- rockstargames.com Entertainment
- zeit.de Other
- expressapisv2.net News & media
- smh.com.au Other
- technologyreview.com Other
- urldefense.com Other
- thehill.com Other
- tmdb.org Other
- gutenberg.org Other
- ko-fi.com Other
- livescience.com Other
- theglobeandmail.com Other
- gamespot.com Entertainment
- hindustantimes.com News & media
- nextdoor.com Social
- huntress.io Other
- cell.com Other
- system-monitor.com Other
- newscientist.com News & media
- franceinfo.fr Other
- salesforce-scrt.com Other
- indianexpress.com News & media
- mastodon.social Other
- bizjournals.com News & media
- realtor.com Other
- faz.net Other
- wa.gov Government
Showing the 200 most popular of 1,134. The full list is in sites.csv.
Sites that block Meta-ExternalAgent on some paths (133)
- soundcloud.com Entertainment
- ubuntu.com Tech
- www.gov.uk Government
- trustpilot.com Other
- tradingview.com News & media
- statista.com News & media
- sohu.com Tech
- trendmicro.com Tech
- squarespace.com Tech
- klaviyo.com Tech
- jotform.com Tech
- airbnb.com Travel
- bestbuy.com E-commerce
- disneyplus.com Entertainment
- expedia.com Travel
- jimdo.com Tech
- maricopa.gov Government
- finn.no E-commerce
- fmkorea.com Social
- amap.com Travel
- jusbrasil.com.br Other
- deezer.com Other
- hostinger.com Tech
- gartner.com Other
- kargo.com Other
- fastcompany.com Other
- mindbox.ru Other
- infomaniak.com Other
- inc.com Other
- canonical.com Other
- gmx.net Other
- vtex.com Other
- bu.edu Education
- vrbo.com Other
- newsbreak.com News & media
- voximplant.com Other
- hootsuite.com Other
- razorpay.com Finance
- gap.com Other
- postman.com News & media
- digitaltrends.com Other
- dashlane.com Other
- g2.com Other
- alltrails.com Other
- foreignpolicy.com Other
- alamy.com Other
- vrt.be Other
- qualys.com Other
- nordstrom.com Other
- ulta.com Other
- pressreader.com News & media
- cengage.com Other
- apkmirror.com Other
- orbitz.com Other
- hostnet.nl Other
- ziprecruiter.com Other
- myheritage.com Other
- getpostman.com News & media
- manageengine.com Other
- qwant.com Other
Showing the 60 most popular of 133. The full list is in sites.csv.
Trend
| Date | Share blocking Meta-ExternalAgent |
|---|---|
| 2026-11-16 | 13.6% |
| 2026-11-15 | 13.6% |
| 2026-11-14 | 13.6% |
| 2026-11-13 | 13.6% |
| 2026-11-12 | 13.6% |
| 2026-11-11 | 13.6% |
| 2026-11-10 | 13.6% |
| 2026-11-09 | 13.5% |
| 2026-11-08 | 13.5% |
| 2026-11-07 | 13.6% |
| 2026-11-06 | 13.5% |
| 2026-11-05 | 13.5% |
| 2026-11-04 | 13.4% |
| 2026-11-03 | 13.3% |
| 2026-11-02 | 13.2% |
| 2026-11-01 | 13.2% |
| 2026-10-31 | 13.3% |
| 2026-10-30 | 13.2% |
| 2026-10-29 | 13.2% |
| 2026-10-28 | 12.6% |
| 2026-10-27 | 11.9% |
| 2026-10-26 | 11.2% |
| 2026-10-25 | 11.3% |
| 2026-10-24 | 11.1% |
| 2026-10-23 | 11% |
| 2026-10-22 | 11% |
| 2026-10-21 | 11.1% |
| 2026-10-20 | 10.4% |
| 2026-10-19 | 10.3% |
| 2026-10-18 | 10.1% |
Latest changes
- unesco.org: blocked → allowed
- gutenberg.org: allowed → blocked
- transfermarkt.com: allowed → blocked
- dawn.com: allowed → blocked
- athome.co.jp: allowed → blocked
- yclients.com: allowed → blocked
- dreamwidth.org: allowed → blocked
- passportindia.gov.in: blocked → allowed
Block or allow Meta-ExternalAgent
User-agent: Meta-ExternalAgent Disallow: /
Replace Disallow: / with Allow: / to let it in. A site that blocks training crawlers but wants to be cited in AI answers usually keeps the search and user-triggered crawlers allowed.
FAQ
- How many websites block Meta-ExternalAgent?
- As of 2026-11-16, 1,134 of the 8,313 popular websites whose robots.txt we could read (13.6%) block Meta-ExternalAgent, Meta's AI training crawler, on the home page; 133 more block it on some paths. Among the top 100 sites: 25.3%.
- What is Meta-ExternalAgent?
- Meta-ExternalAgent is Meta's AI training crawler: it collects pages to train AI models. It is the #5 most blocked of the 29 AI crawlers in the index.
- How do I block Meta-ExternalAgent in robots.txt?
- Add these lines to https://<your-site>/robots.txt: "User-agent: Meta-ExternalAgent" followed by "Disallow: /". To allow it, use "Allow: /" instead. Well-behaved crawlers read robots.txt before fetching pages; firewall rules are needed to stop crawlers that ignore it.
- How do I check whether my site blocks Meta-ExternalAgent?
- Use the free AI crawler checker at tools.yukai.uk/free/ai-crawler-checker: it shows the status of Meta-ExternalAgent and 28 other AI crawlers with the exact robots.txt rule.
Other AI crawlers
Also from Meta: meta-webindexer (5.1%), meta-externalfetcher (6.5%), meta-externalads (2.9%).
- CCBot 16.4%
- Bytespider 16.2%
- GPTBot 15.4%
- ClaudeBot 14.7%
- Applebot-Extended 13.4%
- Google-Extended 13.2%
- Amazonbot 13.2%
- cohere-ai 12.7%
- Google-CloudVertexBot 8.8%
- PerplexityBot 8.2%
- ChatGPT-User 7.4%
- MistralAI-Training 7.1%
- meta-externalfetcher 6.5%
- Perplexity-User 6.1%
- Claude-User 5.9%
- DuckAssistBot 5.9%
- OAI-SearchBot 5.8%
- Claude-SearchBot 5.7%
- MistralAI-User 5.7%
- meta-webindexer 5.1%
- Amzn-SearchBot 4.4%
- Amzn-User 4.4%
- Google-Agent 3.6%
- MistralAI-Index 3.6%
- Google-GeminiNotebook 3.3%
- Applebot 3.2%
- OAI-AdsBot 3%
- meta-externalads 2.9%