Free AI crawler checker
Is your website blocking ChatGPT, Claude or Perplexity by accident? Check which of 29 AI crawlers a site's robots.txt allows or blocks: training bots (GPTBot, ClaudeBot, CCBot), AI search bots (OAI-SearchBot, Claude-SearchBot, PerplexityBot) and the fetchers that run when a user asks a question (ChatGPT-User, Claude-User). Free, no sign-up, no key.
Use it from code or an AI agent
One GET request, JSON back. Put the domain after the path, or pass ?url=:
curl https://tools.yukai.uk/ai-crawlers/example.com
import requests
r = requests.get("https://tools.yukai.uk/ai-crawlers/example.com", timeout=30)
data = r.json()
print(data["verdict"], data["blockedBots"])
const r = await fetch("https://tools.yukai.uk/ai-crawlers/example.com");
const data = await r.json();
console.log(data.verdict, data.blockedBots);
What the response contains
- verdict: one sentence for people, and policy: open, search-only, blocks-search, restrictive, no-robots or unknown.
- aiSearchScore and aiAccessScore (0-100): the share of AI search and of all AI crawlers allowed on the home page.
- allowedBots, partialBots, blockedBots, and per bot the exact robots.txt rule that decided it.
- contentSignals (Cloudflare Content-Signal: search, ai-input, ai-train), llmsTxt and llmsFullTxt, and the noai / noimageai meta and X-Robots-Tag directives.
- recommendedRobotsSnippet: robots.txt lines that re-allow blocked AI search crawlers, ready to paste; suggestedPolicy: a full template that blocks training and allows AI search.
When robots.txt cannot be read (a firewall challenge, a 403, or a CDN error while the home page loads), the policy is unknown rather than a guess. A site that does not resolve or does not answer is an error (HTTP 422), not "blocked".
Free or paid?
The free endpoint checks one site per request and is rate limited. For a list of sites, scheduled checks with alerts when a site changes its rules, or volume from your own code, use the paid versions of the same check:
| Option | Limits | Best for | Price |
|---|---|---|---|
| Free endpoint (this page) | About 10 requests per minute per IP, one site per request | Checking a few sites, AI agents, quick scripts | Free |
| AI Crawler Access Checker API (RapidAPI) | Free plan: 100 requests/month, with a key | Your own code, one call per item, monthly volume | $9.99/month for 3,000 requests |
| Apify Actor | Many items per run, schedules | Hundreds or thousands of domains, competitor lists, change monitoring | $2 per 1,000 sites, pay per result |
AI agents can pay for the Apify Actor with x402 via Apify (USDC, no Apify account).
FAQ
Is the AI crawler checker really free?
Yes. No sign-up and no API key. It is rate limited to about 10 requests per minute per IP address and checks one site per request.
Which AI crawlers does it check?
29 crawlers, including GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-SearchBot, PerplexityBot and Google-Extended, against the site's robots.txt.
How do I allow ChatGPT and Perplexity to find my site?
Use the recommendedRobotsSnippet in the response: robots.txt lines that re-allow the blocked AI search crawlers. Block only the training bots if you want to keep content out of model training.
What does the verdict unknown mean?
robots.txt could not be read, for example because a firewall challenged the request. The checker reports unknown instead of guessing.
How do I check hundreds of sites or get alerts when rules change?
Use the AI Crawler Access Checker on Apify (pay per site, scheduled runs) or the RapidAPI API (free plan with a key). Both are linked in the table above.
Related
- AI Crawler Index: how about 10,000 of the most visited websites treat AI crawlers, updated daily.
- Free URL to Markdown API: any web page or PDF as clean Markdown for LLMs.
- How to check if GPTBot is blocked on my site
- llms.txt: every TidyTools tool, for AI agents.