TidyToolsAI Crawler Index · data as of · updated daily

AI Crawler Index

AI crawler rules of 10,000 popular websites

One page per site: which of 29 AI crawlers (GPTBot, ClaudeBot, PerplexityBot and more) its robots.txt blocks, whether it has an llms.txt, and what changed. Browse by category, by domain suffix or alphabetically. Checked daily, last on 2026-11-16.

By category

By country and domain suffix

Compare all suffixes · Sites with llms.txt

A-Z

The 200 most popular sites