AI crawler · OpenAI · AI search
Which websites block OAI-SearchBot?
As of 2026-11-16, 479 of the 8,313 popular websites whose robots.txt we could read (5.8%) block OAI-SearchBot, OpenAI's AI search crawler, on the home page; 261 more block it on some paths. Among the top 100 sites: 12%.
5.8%
of 8,313 readable sites block OAI-SearchBot
12%
of the top 100
#18
most blocked of 29 AI crawlers
7,573
sites allow it on the home page
Blocked by category
| Category | Sites read | Block OAI-SearchBot |
|---|---|---|
| Social | 51 | 19.6% |
| News & media | 392 | 17.9% |
| AI | 10 | 10% |
| Reference | 43 | 7% |
| Other | 6651 | 5.6% |
| E-commerce | 116 | 3.4% |
| Entertainment | 332 | 3% |
| Finance | 102 | 2.9% |
| Government | 203 | 2% |
| Education | 232 | 1.3% |
| Tech | 162 | 0% |
| Telecom | 6 | 0% |
| Travel | 13 | 0% |
Popular sites that block OAI-SearchBot (479)
- facebook.com Social
- instagram.com Social
- twitter.com Social
- amazon.com E-commerce
- x.com Social
- pinterest.com Social
- tiktok.com Social
- reddit.com Social
- nytimes.com News & media
- bbc.com News & media
- imdb.com Reference
- launchpad.net Other
- weibo.com Social
- claude.ai AI
- cnbc.com News & media
- quora.com Reference
- usatoday.com News & media
- threads.com Social
- telegraph.co.uk News & media
- dailymail.com News & media
- fb.com Other
- cnet.com News & media
- nbcnews.com News & media
- corriere.it News & media
- elmundo.es News & media
- ouest-france.fr News & media
- huffpost.com News & media
- bfmtv.com News & media
- marca.com News & media
- economist.com News & media
- theconversation.com News & media
- ad.nl News & media
- nu.nl News & media
- elconfidencial.com News & media
- gazzetta.it News & media
- hln.be News & media
- themoviedb.org Reference
- spiegel.de News & media
- actu.fr News & media
- sapo.pt News & media
- huffingtonpost.com News & media
- dagbladet.no News & media
- dw.com News & media
- iltalehti.fi News & media
- ekstrabladet.dk News & media
- tjk.org Other
- aka.ms Other
- gartner.com Other
- qcloud.com Other
- zdnet.com News & media
- rocketmoney.dev Other
- adrta.com Other
- scientificamerican.com Other
- wikihow.com Other
- blackhub.team Other
- sedoparking.com Other
- sogou.com Other
- ign.com Other
- salesforceliveagent.com Other
- adsco.re Other
- mashable.com Other
- openssl.org Other
- variety.com Other
- mirror.co.uk Other
- constellix.com Other
- kasserver.com Other
- congress.gov Government
- co.com Other
- tappx.com Other
- rollingstone.com Other
- redditmedia.com Other
- express.co.uk News & media
- hollywoodreporter.com Other
- news.com.au News & media
- zeit.de Other
- expressapisv2.net News & media
- smh.com.au Other
- technologyreview.com Other
- urldefense.com Other
- thehill.com Other
- tmdb.org Other
- gutenberg.org Other
- theglobeandmail.com Other
- nextdoor.com Social
- huntress.io Other
- system-monitor.com Other
- newscientist.com News & media
- franceinfo.fr Other
- salesforce-scrt.com Other
- faz.net Other
- wa.gov Government
- jwpltx.com Other
- chicagotribune.com News & media
- asahi.com Other
- nrk.no Other
- thehindu.com Other
- ctvnews.ca News & media
- france24.com Other
- hellotalk8.com Other
- today.com Other
- yr.no Other
- metro.co.uk Other
- billboard.com Other
- wargaming.net Other
- nzherald.co.nz News & media
- flipboard.com Other
- cuny.edu Education
- lifehacker.com Other
- deadline.com Other
- lenovomm.com Other
- smugmug.com Other
- stuff.co.nz Other
- rs6.net Other
- thefreedictionary.com Other
- codeberg.org Other
- onenote.com Other
- entrust.net Other
- nydailynews.com News & media
- yomiuri.co.jp Other
- gitea.com Other
- sportradarserving.com News & media
- backblazeb2.com Other
- rfi.fr Other
- findagrave.com Other
- wordreference.com Other
- gmarket.co.kr E-commerce
- thestar.com Other
- mainichi.jp Other
- seattletimes.com News & media
- radiofrance.fr Other
- cnki.net Other
- kompas.com Other
- myhostadmin.net Other
- helsinki.fi Other
- 20minutes.fr Other
- paycomonline.net Finance
- feednews.com News & media
- huya.com Other
- doh.pub Other
- liberation.fr Other
- nine.com.au Other
- ezoic.net Other
- boxofficemojo.com Other
- hurriyet.com.tr Other
- telegraaf.nl Other
- channel4.com Other
- dr.dk Other
- msnbc.com Other
- dayforcehcm.com Other
- band.us Other
- ilsole24ore.com Other
- arte.tv Entertainment
- deccanherald.com News & media
- lapresse.ca News & media
- michelin.com Other
- ms.now Other
- theage.com.au Other
- lesechos.fr Other
- chinatimes.com News & media
- expressen.se News & media
- mercurynews.com News & media
- everydayhealth.com Other
- yna.co.kr Other
- gentoo.org Other
- queue-it.net Other
- braintreegateway.com Other
- dn.se Other
- mypurecloud.com Other
- pro32connect.ru Other
- francetvinfo.fr Other
- sankei.com Other
- nj.com Other
- tv2.no Other
- afterpay.com Finance
- mlnba.fun Other
- nzz.ch Other
- jivosite.com Other
- lesoir.be Other
- ilmessaggero.it Other
- milliyet.com.tr Other
- nieuwsblad.be Other
- haaretz.com Other
- news24.com News & media
- sporcle.com Other
- ici.fr Other
- nyt.com Other
- storygize.net Other
- diigo.com Other
- rtbf.be Other
- securedvisit.com Other
- oregonlive.com Other
- letelegramme.fr Other
- lavoixdunord.fr Other
- azcentral.com Other
- denverpost.com News & media
- megaphone.fm Other
- lunalabs.io Other
- rtvslo.si Other
- parismatch.com Other
- hani.co.kr Other
Showing the 200 most popular of 479. The full list is in sites.csv.
Sites that block OAI-SearchBot on some paths (261)
- linkedin.com Social
- github.com Tech
- netflix.com Entertainment
- yahoo.com Tech
- yandex.ru Tech
- samsung.com Tech
- snapchat.com Social
- flickr.com Other
- soundcloud.com Entertainment
- ebay.com E-commerce
- android.com Tech
- ubuntu.com Tech
- reuters.com News & media
- trustpilot.com Other
- tradingview.com News & media
- bloomberg.com News & media
- wsj.com News & media
- paloaltonetworks.com Tech
- sohu.com Tech
- bitrix24.ru Other
- capcut.com Other
- trendmicro.com Tech
- tripadvisor.com Travel
- klaviyo.com Tech
- airbnb.com Travel
- bestbuy.com E-commerce
- fiverr.com E-commerce
- ted.com Entertainment
- disneyplus.com Entertainment
- people.com News & media
- academia.edu Reference
- figma.com Tech
- expedia.com Travel
- otto.de E-commerce
- investopedia.com News & media
- jimdo.com Tech
- theatlantic.com News & media
- maricopa.gov Government
- skroutz.gr E-commerce
- finn.no E-commerce
- aftonbladet.se News & media
- vg.no News & media
- leboncoin.fr E-commerce
- coursera.org Tech
- amap.com Travel
- douyin.com Social
- jusbrasil.com.br Other
- haberler.com News & media
- ilmeteo.it News & media
- sondakika.com News & media
- deezer.com Other
- therapservices.net Other
- shutterstock.com Other
- hostinger.com Tech
- merkur.de Other
- marketwatch.com E-commerce
- oreilly.com Other
- euronews.com News & media
- fastcompany.com Other
- mindbox.ru Other
Showing the 60 most popular of 261. The full list is in sites.csv.
Trend
| Date | Share blocking OAI-SearchBot |
|---|---|
| 2026-11-16 | 5.8% |
| 2026-11-15 | 5.7% |
| 2026-11-14 | 5.7% |
| 2026-11-13 | 5.7% |
| 2026-11-12 | 5.7% |
| 2026-11-11 | 5.7% |
| 2026-11-10 | 5.7% |
| 2026-11-09 | 5.8% |
| 2026-11-08 | 5.7% |
| 2026-11-07 | 5.8% |
| 2026-11-06 | 5.8% |
| 2026-11-05 | 5.8% |
| 2026-11-04 | 5.8% |
| 2026-11-03 | 5.8% |
| 2026-11-02 | 5.7% |
| 2026-11-01 | 5.8% |
| 2026-10-31 | 5.8% |
| 2026-10-30 | 5.8% |
| 2026-10-29 | 5.8% |
| 2026-10-28 | 5.8% |
| 2026-10-27 | 5.8% |
| 2026-10-26 | 5.8% |
| 2026-10-25 | 5.9% |
| 2026-10-24 | 5.8% |
| 2026-10-23 | 5.8% |
| 2026-10-22 | 6% |
| 2026-10-21 | 6.3% |
| 2026-10-20 | 5.7% |
| 2026-10-19 | 5.7% |
| 2026-10-18 | 5.7% |
Latest changes
- gutenberg.org: allowed → blocked
- ura.news: blocked → partial
- athome.co.jp: allowed → blocked
- yclients.com: allowed → blocked
- my-personaltrainer.it: allowed → blocked
- financialpost.com: allowed → blocked
- dreamwidth.org: allowed → blocked
- passportindia.gov.in: blocked → allowed
Block or allow OAI-SearchBot
User-agent: OAI-SearchBot Disallow: /
Replace Disallow: / with Allow: / to let it in. A site that blocks training crawlers but wants to be cited in AI answers usually keeps the search and user-triggered crawlers allowed.
FAQ
- How many websites block OAI-SearchBot?
- As of 2026-11-16, 479 of the 8,313 popular websites whose robots.txt we could read (5.8%) block OAI-SearchBot, OpenAI's AI search crawler, on the home page; 261 more block it on some paths. Among the top 100 sites: 12%.
- What is OAI-SearchBot?
- OAI-SearchBot is OpenAI's AI search crawler: it indexes pages so an AI search product can find and cite them. It is the #18 most blocked of the 29 AI crawlers in the index.
- How do I block OAI-SearchBot in robots.txt?
- Add these lines to https://<your-site>/robots.txt: "User-agent: OAI-SearchBot" followed by "Disallow: /". To allow it, use "Allow: /" instead. Well-behaved crawlers read robots.txt before fetching pages; firewall rules are needed to stop crawlers that ignore it.
- How do I check whether my site blocks OAI-SearchBot?
- Use the free AI crawler checker at tools.yukai.uk/free/ai-crawler-checker: it shows the status of OAI-SearchBot and 28 other AI crawlers with the exact robots.txt rule.
Other AI crawlers
Also from OpenAI: GPTBot (15.4%), ChatGPT-User (7.4%), OAI-AdsBot (3%).
- CCBot 16.4%
- Bytespider 16.2%
- GPTBot 15.4%
- ClaudeBot 14.7%
- Meta-ExternalAgent 13.6%
- Applebot-Extended 13.4%
- Google-Extended 13.2%
- Amazonbot 13.2%
- cohere-ai 12.7%
- Google-CloudVertexBot 8.8%
- PerplexityBot 8.2%
- ChatGPT-User 7.4%
- MistralAI-Training 7.1%
- meta-externalfetcher 6.5%
- Perplexity-User 6.1%
- Claude-User 5.9%
- DuckAssistBot 5.9%
- Claude-SearchBot 5.7%
- MistralAI-User 5.7%
- meta-webindexer 5.1%
- Amzn-SearchBot 4.4%
- Amzn-User 4.4%
- Google-Agent 3.6%
- MistralAI-Index 3.6%
- Google-GeminiNotebook 3.3%
- Applebot 3.2%
- OAI-AdsBot 3%
- meta-externalads 2.9%