How to check if GPTBot is blocked on my site
Open https://tools.yukai.uk/ai-crawlers/yourdomain.com (free, no sign-up) or run the AI Crawler Access Checker on Apify for up to 10,000 domains: each site gets GPTBot's status (allowed, partly blocked, blocked or unknown) from its robots.txt, the rule that matched and a ready-to-paste fix, for $2 per 1,000 sites on Apify.
Last updated
How to do it in 5 steps
- Open
https://tools.yukai.uk/ai-crawlers/example.comwith your own domain instead of example.com, or use the form on the free AI crawler checker page. - See whether GPTBot is listed under Blocked, Partly blocked, Allowed or Unknown (unknown means the site's robots.txt could not be read).
- Check OAI-SearchBot and ChatGPT-User too: GPTBot only collects training data, while ChatGPT search uses OAI-SearchBot and ChatGPT-User fetches pages when a person asks.
- To change the result, paste the ready-made lines from the JSON result (
recommendedRobotsSnippet) into your robots.txt;suggestedPolicyis a full template that blocks training crawlers and allows AI search crawlers. - For many sites, or to be told when a site changes its rules, run the Apify Actor with your list and a monitor name on a schedule.
Specifics
| Crawlers checked | 29 AI crawlers from OpenAI, Anthropic, Perplexity, Google, Apple, Meta, Amazon, DuckDuckGo, Mistral, Common Crawl, ByteDance and Cohere (list checked against each vendor's documentation on 2026-10-01) |
|---|---|
| Free check | https://tools.yukai.uk/ai-crawlers/<domain>: JSON, no key, CORS open, about 10 requests per minute per IP |
| Price on Apify | $2 per 1,000 sites; an unchanged site in a weekly monitor is $0.50 per 1,000; no start fee |
| Not charged | Unreachable sites and sites whose robots.txt cannot be read (401, 403, 429, bot challenge, CDN error) |
| What it does not see | Firewall or CDN rules that block a bot even when robots.txt allows it; GEO Readiness Audit runs that live test (about $0.018 per site) |
| RapidAPI plans | free 100 requests/month; $9.99/month for 3,000 requests |
FAQ
Will blocking GPTBot remove my site from ChatGPT answers?
No. GPTBot collects training data. ChatGPT search uses OAI-SearchBot, and ChatGPT-User fetches pages when a person asks. You can block GPTBot and still allow the other two.
What does partial mean?
The home page is allowed, but one of the tested paths is blocked, or the bot has its own group in robots.txt with Disallow rules.
robots.txt allows AI search. Can the site still block AI crawlers?
Yes. A CDN or firewall rule can block them regardless of robots.txt. The checker reads robots.txt rules only; GEO Readiness Audit runs a live firewall test.
How do I block AI training but keep AI search traffic?
Block the training crawlers (GPTBot, ClaudeBot, Google-Extended, CCBot) and allow the search and assistant crawlers (OAI-SearchBot, ChatGPT-User, Claude-SearchBot, PerplexityBot). Every result includes this template as suggestedPolicy.
Related guides
Related
- Free AI crawler checker (form and JSON API)
- Which popular websites block GPTBot? (daily index)
- All answers
Crawler list, prices and limits from the AI Crawler Access Checker README on Apify and the free endpoint's documentation on this site. Prices are the base (Free plan) prices on Apify; larger Apify plans pay less. Actor page: apify.com/tidytools/ai-crawler-access-checker.