TidyToolsAI Crawler Index · data as of · updated daily

AI crawler · DuckDuckGo · AI search

Which websites block DuckAssistBot?

As of 2026-10-01, 69 of the 853 popular websites whose robots.txt we could read (8.1%) block DuckAssistBot, DuckDuckGo's AI search crawler, on the home page; 20 more block it on some paths. Among the top 100 sites: 15.7%.

8.1%
of 853 readable sites block DuckAssistBot
15.7%
of the top 100
#17
most blocked of 29 AI crawlers
764
sites allow it on the home page

Blocked by category

CategorySites readBlock DuckAssistBot
News & media12623.8%
Social4818.8%
Other2608.1%
Travel137.7%
E-commerce585.2%
Entertainment684.4%
Reference432.3%
Tech1540.6%
AI90%
Education200%
Finance160%
Government330%
Telecom50%

Popular sites that block DuckAssistBot (69)

  1. facebook.com Social
  2. instagram.com Social
  3. twitter.com Social
  4. linkedin.com Social
  5. amazon.com E-commerce
  6. netflix.com Entertainment
  7. x.com Social
  8. tiktok.com Social
  9. reddit.com Social
  10. amazonvideo.com Entertainment
  11. nytimes.com News & media
  12. flickr.com Other
  13. theguardian.com News & media
  14. meraki.com Tech
  15. launchpad.net Other
  16. reuters.com News & media
  17. weibo.com Social
  18. trustpilot.com Other
  19. bloomberg.com News & media
  20. wsj.com News & media
  21. wired.com News & media
  22. uol.com.br News & media
  23. usatoday.com News & media
  24. threads.com Social
  25. telegraph.co.uk News & media
  26. primevideo.com Entertainment
  27. elpais.com News & media
  28. dailymail.com News & media
  29. lemonde.fr News & media
  30. fb.com Other
  31. cnet.com News & media
  32. bild.de News & media
  33. welt.de News & media
  34. people.com News & media
  35. theverge.com News & media
  36. repubblica.it News & media
  37. investopedia.com News & media
  38. flashscore.com News & media
  39. as.com News & media
  40. huffpost.com News & media
  41. theconversation.com News & media
  42. amap.com Travel
  43. emag.ro E-commerce
  44. themoviedb.org Reference
  45. huffingtonpost.com News & media
  46. sapo.pt News & media
  47. iltalehti.fi News & media
  48. ekstrabladet.dk News & media
  49. tjk.org Other
  50. buzzfeed.com Other
  51. aka.ms Other
  52. sportfilm800.com News & media
  53. rocketmoney.dev Other
  54. qcloud.com Other
  55. newyorker.com Other
  56. blackhub.team Other
  57. arstechnica.com News & media
  58. zdnet.com News & media
  59. adrta.com Other
  60. marketwatch.com E-commerce
  61. merkur.de Other
  62. wikihow.com Other
  63. hola.com Other
  64. fastcompany.com Other
  65. lnkd.in Other
  66. sogou.com Other
  67. ign.com Other
  68. salesforceliveagent.com Other
  69. sedoparking.com Other

Sites that block DuckAssistBot on some paths (20)

  1. pinterest.com Social
  2. snapchat.com Social
  3. imdb.com Reference
  4. hostinger.com Tech
  5. indeed.com Other
  6. sohu.com Tech
  7. statista.com News & media
  8. capcut.com Other
  9. trendmicro.com Tech
  10. squarespace.com Tech
  11. klaviyo.com Tech
  12. ted.com Entertainment
  13. disneyplus.com Entertainment
  14. academia.edu Reference
  15. jimdo.com Tech
  16. leboncoin.fr E-commerce
  17. deezer.com Other
  18. kargo.com Other
  19. hulu.com Entertainment
  20. oreilly.com Other

Trend

The daily series for DuckAssistBot starts on 2026-10-01; a trend appears here as days accumulate.

Latest changes

Block or allow DuckAssistBot

User-agent: DuckAssistBot
Disallow: /

Replace Disallow: / with Allow: / to let it in. A site that blocks training crawlers but wants to be cited in AI answers usually keeps the search and user-triggered crawlers allowed.

FAQ

How many websites block DuckAssistBot?
As of 2026-10-01, 69 of the 853 popular websites whose robots.txt we could read (8.1%) block DuckAssistBot, DuckDuckGo's AI search crawler, on the home page; 20 more block it on some paths. Among the top 100 sites: 15.7%.
What is DuckAssistBot?
DuckAssistBot is DuckDuckGo's AI search crawler: it indexes pages so an AI search product can find and cite them. It is the #17 most blocked of the 29 AI crawlers in the index.
How do I block DuckAssistBot in robots.txt?
Add these lines to https://<your-site>/robots.txt: "User-agent: DuckAssistBot" followed by "Disallow: /". To allow it, use "Allow: /" instead. Well-behaved crawlers read robots.txt before fetching pages; firewall rules are needed to stop crawlers that ignore it.
How do I check whether my site blocks DuckAssistBot?
Use the free AI crawler checker at tools.yukai.uk/free/ai-crawler-checker: it shows the status of DuckAssistBot and 28 other AI crawlers with the exact robots.txt rule.

Other AI crawlers