TidyToolsAI Crawler Index · data as of · updated daily

AI crawler · Meta · Training

Which websites block Meta-ExternalAgent?

As of 2026-11-16, 1,134 of the 8,313 popular websites whose robots.txt we could read (13.6%) block Meta-ExternalAgent, Meta's AI training crawler, on the home page; 133 more block it on some paths. Among the top 100 sites: 25.3%.

13.6%
of 8,313 readable sites block Meta-ExternalAgent
25.3%
of the top 100
#5
most blocked of 29 AI crawlers
7,046
sites allow it on the home page

Blocked by category

CategorySites readBlock Meta-ExternalAgent
Social5133.3%
News & media39230.6%
Entertainment33219%
Reference4318.6%
E-commerce11614.7%
Other665113.3%
Travel137.7%
Finance1026.9%
Education2323.9%
Government2032.5%
Tech1621.2%
AI100%
Telecom60%

Popular sites that block Meta-ExternalAgent (1,134)

Showing the 200 most popular of 1,134. The full list is in sites.csv.

Sites that block Meta-ExternalAgent on some paths (133)

Showing the 60 most popular of 133. The full list is in sites.csv.

Trend

DateShare blocking Meta-ExternalAgent
2026-11-1613.6%
2026-11-1513.6%
2026-11-1413.6%
2026-11-1313.6%
2026-11-1213.6%
2026-11-1113.6%
2026-11-1013.6%
2026-11-0913.5%
2026-11-0813.5%
2026-11-0713.6%
2026-11-0613.5%
2026-11-0513.5%
2026-11-0413.4%
2026-11-0313.3%
2026-11-0213.2%
2026-11-0113.2%
2026-10-3113.3%
2026-10-3013.2%
2026-10-2913.2%
2026-10-2812.6%
2026-10-2711.9%
2026-10-2611.2%
2026-10-2511.3%
2026-10-2411.1%
2026-10-2311%
2026-10-2211%
2026-10-2111.1%
2026-10-2010.4%
2026-10-1910.3%
2026-10-1810.1%

Latest changes

Block or allow Meta-ExternalAgent

User-agent: Meta-ExternalAgent
Disallow: /

Replace Disallow: / with Allow: / to let it in. A site that blocks training crawlers but wants to be cited in AI answers usually keeps the search and user-triggered crawlers allowed.

FAQ

How many websites block Meta-ExternalAgent?
As of 2026-11-16, 1,134 of the 8,313 popular websites whose robots.txt we could read (13.6%) block Meta-ExternalAgent, Meta's AI training crawler, on the home page; 133 more block it on some paths. Among the top 100 sites: 25.3%.
What is Meta-ExternalAgent?
Meta-ExternalAgent is Meta's AI training crawler: it collects pages to train AI models. It is the #5 most blocked of the 29 AI crawlers in the index.
How do I block Meta-ExternalAgent in robots.txt?
Add these lines to https://<your-site>/robots.txt: "User-agent: Meta-ExternalAgent" followed by "Disallow: /". To allow it, use "Allow: /" instead. Well-behaved crawlers read robots.txt before fetching pages; firewall rules are needed to stop crawlers that ignore it.
How do I check whether my site blocks Meta-ExternalAgent?
Use the free AI crawler checker at tools.yukai.uk/free/ai-crawler-checker: it shows the status of Meta-ExternalAgent and 28 other AI crawlers with the exact robots.txt rule.

Other AI crawlers

Also from Meta: meta-webindexer (5.1%), meta-externalfetcher (6.5%), meta-externalads (2.9%).