AI crawler access check API: /api/v1/ai-crawlers

Check which AI crawlers a site allows. Reads robots.txt (RFC 9309 matching, with vendor-documented fallbacks such as Applebot following Googlebot), llms.txt, llms-full.txt and the home page's robots meta tags.

Cost

Credits: 18 ($0.0018) per successful call. Failed calls are free. Source price: ai-crawler-access-checker/site-checked.

Input

Send a JSON body (POST) or the same fields as a query string (GET).

Required: url.

Field	Type	Default	Description
`url` required	string		Website or page (a bare domain works). A path in the URL is also tested. Alias: `domain`.
`paths`	string		Up to 5 extra paths to test, comma-separated (e.g. `/blog/,/docs/`).
`includeSnippets`	boolean	`true`	Include the robots.txt fix snippet and a suggested AI policy.

Examples

Replace tt_live_XXXX with your key from the pricing page.

curl

curl -i -X POST "https://tools.yukai.uk/api/v1/ai-crawlers" \
  -H "Authorization: Bearer tt_live_XXXX" \
  -H "Content-Type: application/json" \
  -d '{"url":"nytimes.com"}'

JavaScript (fetch)

const res = await fetch("https://tools.yukai.uk/api/v1/ai-crawlers", {
  method: "POST",
  headers: { "Authorization": "Bearer tt_live_XXXX", "Content-Type": "application/json" },
  body: JSON.stringify({"url":"nytimes.com"}),
});
console.log(res.status, res.headers.get("x-credits-charged"), res.headers.get("x-credits-balance"));
console.log(await res.json());

Python (requests)

import requests

r = requests.post(
    "https://tools.yukai.uk/api/v1/ai-crawlers",
    headers={"Authorization": "Bearer tt_live_XXXX"},
    json={"url": "nytimes.com"},
    timeout=120,
)
print(r.status_code, r.headers.get("x-credits-charged"), r.headers.get("x-credits-balance"))
print(r.json())

Response

JSON with "success": true and the result fields; on failure {"success": false, "error": {"code", "message", "retryable"}} and no credits are used. Every answer carries x-credits-charged (credits used) and x-credits-balance (balance after the call); check the balance any time with GET https://tools.yukai.uk/v1/billing/balance.

Full field list and response schema: OpenAPI spec ai-crawler-access (its servers entry is the marketplace gateway; call https://tools.yukai.uk with your key).

← All routes · Pricing