TidyToolsAI Crawler Index · data as of · updated daily

AI crawler · OpenAI · AI search

Which websites block OAI-SearchBot?

As of 2026-11-16, 479 of the 8,313 popular websites whose robots.txt we could read (5.8%) block OAI-SearchBot, OpenAI's AI search crawler, on the home page; 261 more block it on some paths. Among the top 100 sites: 12%.

5.8%
of 8,313 readable sites block OAI-SearchBot
12%
of the top 100
#18
most blocked of 29 AI crawlers
7,573
sites allow it on the home page

Blocked by category

CategorySites readBlock OAI-SearchBot
Social5119.6%
News & media39217.9%
AI1010%
Reference437%
Other66515.6%
E-commerce1163.4%
Entertainment3323%
Finance1022.9%
Government2032%
Education2321.3%
Tech1620%
Telecom60%
Travel130%

Popular sites that block OAI-SearchBot (479)

Showing the 200 most popular of 479. The full list is in sites.csv.

Sites that block OAI-SearchBot on some paths (261)

Showing the 60 most popular of 261. The full list is in sites.csv.

Trend

DateShare blocking OAI-SearchBot
2026-11-165.8%
2026-11-155.7%
2026-11-145.7%
2026-11-135.7%
2026-11-125.7%
2026-11-115.7%
2026-11-105.7%
2026-11-095.8%
2026-11-085.7%
2026-11-075.8%
2026-11-065.8%
2026-11-055.8%
2026-11-045.8%
2026-11-035.8%
2026-11-025.7%
2026-11-015.8%
2026-10-315.8%
2026-10-305.8%
2026-10-295.8%
2026-10-285.8%
2026-10-275.8%
2026-10-265.8%
2026-10-255.9%
2026-10-245.8%
2026-10-235.8%
2026-10-226%
2026-10-216.3%
2026-10-205.7%
2026-10-195.7%
2026-10-185.7%

Latest changes

Block or allow OAI-SearchBot

User-agent: OAI-SearchBot
Disallow: /

Replace Disallow: / with Allow: / to let it in. A site that blocks training crawlers but wants to be cited in AI answers usually keeps the search and user-triggered crawlers allowed.

FAQ

How many websites block OAI-SearchBot?
As of 2026-11-16, 479 of the 8,313 popular websites whose robots.txt we could read (5.8%) block OAI-SearchBot, OpenAI's AI search crawler, on the home page; 261 more block it on some paths. Among the top 100 sites: 12%.
What is OAI-SearchBot?
OAI-SearchBot is OpenAI's AI search crawler: it indexes pages so an AI search product can find and cite them. It is the #18 most blocked of the 29 AI crawlers in the index.
How do I block OAI-SearchBot in robots.txt?
Add these lines to https://<your-site>/robots.txt: "User-agent: OAI-SearchBot" followed by "Disallow: /". To allow it, use "Allow: /" instead. Well-behaved crawlers read robots.txt before fetching pages; firewall rules are needed to stop crawlers that ignore it.
How do I check whether my site blocks OAI-SearchBot?
Use the free AI crawler checker at tools.yukai.uk/free/ai-crawler-checker: it shows the status of OAI-SearchBot and 28 other AI crawlers with the exact robots.txt rule.

Other AI crawlers

Also from OpenAI: GPTBot (15.4%), ChatGPT-User (7.4%), OAI-AdsBot (3%).