Free AI crawler checker
Is your website blocking ChatGPT, Claude or Perplexity by accident? Check which of 29 AI crawlers a site's robots.txt allows or blocks: training bots (GPTBot, ClaudeBot, CCBot), AI search bots (OAI-SearchBot, Claude-SearchBot, PerplexityBot) and the fetchers that run when a user asks a question (ChatGPT-User, Claude-User). Free, no sign-up, no key.
Use it from code or an AI agent
One GET request, JSON back. Put the domain after the path, or pass ?url=:
curl https://tools.yukai.uk/ai-crawlers/example.comimport requests
r = requests.get("https://tools.yukai.uk/ai-crawlers/example.com", timeout=30)
data = r.json()
print(data["verdict"], data["blockedBots"])const r = await fetch("https://tools.yukai.uk/ai-crawlers/example.com");
const data = await r.json();
console.log(data.verdict, data.blockedBots);What the response contains
- verdict: one sentence for people, and policy: open, search-only, blocks-search, restrictive, no-robots or unknown.
- aiSearchScore and aiAccessScore (0-100): the share of AI search and of all AI crawlers allowed on the home page.
- allowedBots, partialBots, blockedBots, and per bot the exact robots.txt rule that decided it.
- contentSignals (Cloudflare Content-Signal: search, ai-input, ai-train), llmsTxt and llmsFullTxt, and the noai / noimageai meta and X-Robots-Tag directives.
- recommendedRobotsSnippet: robots.txt lines that re-allow blocked AI search crawlers, ready to paste; suggestedPolicy: a full template that blocks training and allows AI search.
When robots.txt cannot be read (a firewall challenge, a 403, or a CDN error while the home page loads), the policy is unknown rather than a guess. A site that does not resolve or does not answer is an error (HTTP 422), not "blocked".
Free or paid?
The free endpoint checks one site per request and is rate limited. For a list of sites, scheduled checks with alerts when a site changes its rules, or volume from your own code, use the paid versions of the same check:
| Option | Limits | Best for | Price |
|---|---|---|---|
| Free endpoint (this page) | About 10 requests per minute per IP, one site per request | Checking a few sites, AI agents, quick scripts | Free |
| AI Crawler Access Checker API (RapidAPI) | Free plan: 100 requests/month, with a key | Your own code, one call per item, monthly volume | $9.99/month for 3,000 requests |
| Apify Actor | Many items per run, schedules | Hundreds or thousands of domains, competitor lists, change monitoring | $2 per 1,000 sites, pay per result |
Related
- AI Crawler Index: how about 1,000 of the most visited websites treat AI crawlers, updated daily.
- Free URL to Markdown API: any web page or PDF as clean Markdown for LLMs.
- llms.txt: every TidyTools tool, for AI agents.