TidyToolsAI Crawler Index · data as of · updated daily

AI crawler · OpenAI · AI search

Which websites block OAI-SearchBot?

As of 2026-10-01, 57 of the 853 popular websites whose robots.txt we could read (6.7%) block OAI-SearchBot, OpenAI's AI search crawler, on the home page; 61 more block it on some paths. Among the top 100 sites: 12%.

6.7%
of 853 readable sites block OAI-SearchBot
12%
of the top 100
#19
most blocked of 29 AI crawlers
735
sites allow it on the home page

Blocked by category

CategorySites readBlock OAI-SearchBot
News & media12621.4%
Social4818.8%
AI911.1%
Reference437%
Other2605.4%
E-commerce583.4%
Tech1540.6%
Education200%
Entertainment680%
Finance160%
Government330%
Telecom50%
Travel130%

Popular sites that block OAI-SearchBot (57)

  1. facebook.com Social
  2. instagram.com Social
  3. twitter.com Social
  4. amazon.com E-commerce
  5. x.com Social
  6. pinterest.com Social
  7. tiktok.com Social
  8. reddit.com Social
  9. nytimes.com News & media
  10. bbc.com News & media
  11. meraki.com Tech
  12. imdb.com Reference
  13. launchpad.net Other
  14. weibo.com Social
  15. claude.ai AI
  16. cnbc.com News & media
  17. quora.com Reference
  18. usatoday.com News & media
  19. threads.com Social
  20. telegraph.co.uk News & media
  21. dailymail.com News & media
  22. fb.com Other
  23. cnet.com News & media
  24. nbcnews.com News & media
  25. elmundo.es News & media
  26. bfmtv.com News & media
  27. huffpost.com News & media
  28. marca.com News & media
  29. economist.com News & media
  30. theconversation.com News & media
  31. ad.nl News & media
  32. nu.nl News & media
  33. elconfidencial.com News & media
  34. spiegel.de News & media
  35. hln.be News & media
  36. emag.ro E-commerce
  37. themoviedb.org Reference
  38. actu.fr News & media
  39. huffingtonpost.com News & media
  40. sapo.pt News & media
  41. dw.com News & media
  42. iltalehti.fi News & media
  43. ekstrabladet.dk News & media
  44. tjk.org Other
  45. aka.ms Other
  46. sportfilm800.com News & media
  47. rocketmoney.dev Other
  48. qcloud.com Other
  49. blackhub.team Other
  50. zdnet.com News & media
  51. adrta.com Other
  52. scientificamerican.com Other
  53. wikihow.com Other
  54. sogou.com Other
  55. ign.com Other
  56. salesforceliveagent.com Other
  57. sedoparking.com Other

Sites that block OAI-SearchBot on some paths (61)

  1. linkedin.com Social
  2. github.com Tech
  3. netflix.com Entertainment
  4. yahoo.com Tech
  5. samsung.com Tech
  6. snapchat.com Social
  7. flickr.com Other
  8. soundcloud.com Entertainment
  9. ebay.com E-commerce
  10. oracle.com Tech
  11. android.com Tech
  12. ubuntu.com Tech
  13. reuters.com News & media
  14. hostinger.com Tech
  15. trustpilot.com Other
  16. tradingview.com News & media
  17. bloomberg.com News & media
  18. wsj.com News & media
  19. indeed.com Other
  20. temu.com E-commerce
  21. paloaltonetworks.com Tech
  22. sohu.com Tech
  23. capcut.com Other
  24. bitrix24.ru Other
  25. trendmicro.com Tech
  26. tripadvisor.com Travel
  27. klaviyo.com Tech
  28. airbnb.com Travel
  29. bestbuy.com E-commerce
  30. ted.com Entertainment
  31. disneyplus.com Entertainment
  32. people.com News & media
  33. academia.edu Reference
  34. figma.com Tech
  35. expedia.com Travel
  36. otto.de E-commerce
  37. investopedia.com News & media
  38. jimdo.com Tech
  39. skroutz.gr E-commerce
  40. finn.no E-commerce
  41. aftonbladet.se News & media
  42. gsmarena.com News & media
  43. vg.no News & media
  44. leboncoin.fr E-commerce
  45. coursera.org Tech
  46. amap.com Travel
  47. douyin.com Social
  48. bbb.org Other
  49. dagbladet.no News & media
  50. ilmeteo.it News & media
  51. deezer.com Other
  52. therapservices.net Other
  53. shutterstock.com Other
  54. marketwatch.com E-commerce
  55. merkur.de Other
  56. hulu.com Entertainment
  57. oreilly.com Other
  58. hh.ru Other
  59. euronews.com News & media
  60. fastcompany.com Other

Showing the 60 most popular of 61. The full list is in sites.csv.

Trend

The daily series for OAI-SearchBot starts on 2026-10-01; a trend appears here as days accumulate.

Latest changes

Block or allow OAI-SearchBot

User-agent: OAI-SearchBot
Disallow: /

Replace Disallow: / with Allow: / to let it in. A site that blocks training crawlers but wants to be cited in AI answers usually keeps the search and user-triggered crawlers allowed.

FAQ

How many websites block OAI-SearchBot?
As of 2026-10-01, 57 of the 853 popular websites whose robots.txt we could read (6.7%) block OAI-SearchBot, OpenAI's AI search crawler, on the home page; 61 more block it on some paths. Among the top 100 sites: 12%.
What is OAI-SearchBot?
OAI-SearchBot is OpenAI's AI search crawler: it indexes pages so an AI search product can find and cite them. It is the #19 most blocked of the 29 AI crawlers in the index.
How do I block OAI-SearchBot in robots.txt?
Add these lines to https://<your-site>/robots.txt: "User-agent: OAI-SearchBot" followed by "Disallow: /". To allow it, use "Allow: /" instead. Well-behaved crawlers read robots.txt before fetching pages; firewall rules are needed to stop crawlers that ignore it.
How do I check whether my site blocks OAI-SearchBot?
Use the free AI crawler checker at tools.yukai.uk/free/ai-crawler-checker: it shows the status of OAI-SearchBot and 28 other AI crawlers with the exact robots.txt rule.

Other AI crawlers

Also from OpenAI: GPTBot (17%), ChatGPT-User (9.6%), OAI-AdsBot (4.3%).