TidyTools

Free AI crawler checker

Is your website blocking ChatGPT, Claude or Perplexity by accident? Check which of 29 AI crawlers a site's robots.txt allows or blocks: training bots (GPTBot, ClaudeBot, CCBot), AI search bots (OAI-SearchBot, Claude-SearchBot, PerplexityBot) and the fetchers that run when a user asks a question (ChatGPT-User, Claude-User). Free, no sign-up, no key.

Use it from code or an AI agent

One GET request, JSON back. Put the domain after the path, or pass ?url=:

curl https://tools.yukai.uk/ai-crawlers/example.com
import requests

r = requests.get("https://tools.yukai.uk/ai-crawlers/example.com", timeout=30)
data = r.json()
print(data["verdict"], data["blockedBots"])
const r = await fetch("https://tools.yukai.uk/ai-crawlers/example.com");
const data = await r.json();
console.log(data.verdict, data.blockedBots);

What the response contains

When robots.txt cannot be read (a firewall challenge, a 403, or a CDN error while the home page loads), the policy is unknown rather than a guess. A site that does not resolve or does not answer is an error (HTTP 422), not "blocked".

Free or paid?

The free endpoint checks one site per request and is rate limited. For a list of sites, scheduled checks with alerts when a site changes its rules, or volume from your own code, use the paid versions of the same check:

OptionLimitsBest forPrice
Free endpoint (this page)About 10 requests per minute per IP, one site per requestChecking a few sites, AI agents, quick scriptsFree
AI Crawler Access Checker API (RapidAPI)Free plan: 100 requests/month, with a keyYour own code, one call per item, monthly volume$9.99/month for 3,000 requests
Apify ActorMany items per run, schedulesHundreds or thousands of domains, competitor lists, change monitoring$2 per 1,000 sites, pay per result

AI agents can pay for the Apify Actor with x402 via Apify (USDC, no Apify account).

FAQ

Is the AI crawler checker really free?

Yes. No sign-up and no API key. It is rate limited to about 10 requests per minute per IP address and checks one site per request.

Which AI crawlers does it check?

29 crawlers, including GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-SearchBot, PerplexityBot and Google-Extended, against the site's robots.txt.

How do I allow ChatGPT and Perplexity to find my site?

Use the recommendedRobotsSnippet in the response: robots.txt lines that re-allow the blocked AI search crawlers. Block only the training bots if you want to keep content out of model training.

What does the verdict unknown mean?

robots.txt could not be read, for example because a firewall challenged the request. The checker reports unknown instead of guessing.

How do I check hundreds of sites or get alerts when rules change?

Use the AI Crawler Access Checker on Apify (pay per site, scheduled runs) or the RapidAPI API (free plan with a key). Both are linked in the table above.

Related