TidyToolsAI Crawler Index · data as of · updated daily

AI crawler · Meta · AI search

Which websites block meta-webindexer?

As of 2026-10-01, 54 of the 853 popular websites whose robots.txt we could read (6.3%) block meta-webindexer, Meta's AI search crawler, on the home page; 7 more block it on some paths. Among the top 100 sites: 14.5%.

6.3%
of 853 readable sites block meta-webindexer
14.5%
of the top 100
#20
most blocked of 29 AI crawlers
792
sites allow it on the home page

Blocked by category

CategorySites readBlock meta-webindexer
Social4818.8%
News & media12611.9%
Travel137.7%
Entertainment687.4%
Other2606.5%
E-commerce585.2%
Reference434.7%
Tech1541.3%
AI90%
Education200%
Finance160%
Government330%
Telecom50%

Popular sites that block meta-webindexer (54)

  1. facebook.com Social
  2. instagram.com Social
  3. twitter.com Social
  4. linkedin.com Social
  5. amazon.com E-commerce
  6. netflix.com Entertainment
  7. x.com Social
  8. pinterest.com Social
  9. reddit.com Social
  10. amazonvideo.com Entertainment
  11. nytimes.com News & media
  12. flickr.com Other
  13. meraki.com Tech
  14. imdb.com Reference
  15. weibo.com Social
  16. trustpilot.com Other
  17. wsj.com News & media
  18. sohu.com Tech
  19. wired.com News & media
  20. usatoday.com News & media
  21. threads.com Social
  22. primevideo.com Entertainment
  23. lemonde.fr News & media
  24. fb.com Other
  25. cnet.com News & media
  26. disneyplus.com Entertainment
  27. people.com News & media
  28. academia.edu Reference
  29. onet.pl News & media
  30. investopedia.com News & media
  31. bfmtv.com News & media
  32. theconversation.com News & media
  33. amap.com Travel
  34. emag.ro E-commerce
  35. iltalehti.fi News & media
  36. tjk.org Other
  37. aka.ms Other
  38. sportfilm800.com News & media
  39. rocketmoney.dev Other
  40. qcloud.com Other
  41. newyorker.com Other
  42. blackhub.team Other
  43. arstechnica.com News & media
  44. zdnet.com News & media
  45. adrta.com Other
  46. marketwatch.com E-commerce
  47. merkur.de Other
  48. hulu.com Entertainment
  49. wikihow.com Other
  50. lnkd.in Other
  51. sogou.com Other
  52. ign.com Other
  53. salesforceliveagent.com Other
  54. sedoparking.com Other

Sites that block meta-webindexer on some paths (7)

  1. reuters.com News & media
  2. indeed.com Other
  3. capcut.com Other
  4. deezer.com Other
  5. oreilly.com Other
  6. autotrader.co.uk Other
  7. fastcompany.com Other

Trend

The daily series for meta-webindexer starts on 2026-10-01; a trend appears here as days accumulate.

Latest changes

Block or allow meta-webindexer

User-agent: meta-webindexer
Disallow: /

Replace Disallow: / with Allow: / to let it in. A site that blocks training crawlers but wants to be cited in AI answers usually keeps the search and user-triggered crawlers allowed.

FAQ

How many websites block meta-webindexer?
As of 2026-10-01, 54 of the 853 popular websites whose robots.txt we could read (6.3%) block meta-webindexer, Meta's AI search crawler, on the home page; 7 more block it on some paths. Among the top 100 sites: 14.5%.
What is meta-webindexer?
meta-webindexer is Meta's AI search crawler: it indexes pages so an AI search product can find and cite them. It is the #20 most blocked of the 29 AI crawlers in the index.
How do I block meta-webindexer in robots.txt?
Add these lines to https://<your-site>/robots.txt: "User-agent: meta-webindexer" followed by "Disallow: /". To allow it, use "Allow: /" instead. Well-behaved crawlers read robots.txt before fetching pages; firewall rules are needed to stop crawlers that ignore it.
How do I check whether my site blocks meta-webindexer?
Use the free AI crawler checker at tools.yukai.uk/free/ai-crawler-checker: it shows the status of meta-webindexer and 28 other AI crawlers with the exact robots.txt rule.

Other AI crawlers

Also from Meta: Meta-ExternalAgent (14.4%), meta-externalfetcher (8.7%), meta-externalads (3.8%).