Who blocks AI crawlers?

Key findings

• Training is blocked more than search. 15.3% of sites block OpenAI's training crawler GPTBot, but only 5.7% block OAI-SearchBot, the crawler behind ChatGPT search.
• Most blocked: CCBot (16.3%), Bytespider (16.1%), GPTBot (15.3%).
• Social sites are the most restrictive: 34% block ClaudeBot and 36% block GPTBot. Tech is the most open (3.7% block GPTBot; categories with 20+ sites).
• The biggest sites block more: 22.9% of the top 100 block GPTBot, versus 15.3% across all 10,000.
• ClaudeBot is blocked about as often as GPTBot, PerplexityBot less often: ClaudeBot 14.6%, PerplexityBot 8.2%, GPTBot 15.3%.
• llms.txt is still rare: 1,391 of 10,000 sites publish one; 502 use Cloudflare's Content-Signal in robots.txt (411 of them say ai-train=no).

Block rate by crawler

• Training
• AI search
• User-triggered fetch
• Ads review

By category

Share of readable sites in each category that block the crawler on the home page. Sorted by average block rate.

Policy mix

• Open 5,511 (55.1%)
  No listed AI crawler is blocked on the home page
• Blocks training only 514 (5.1%)
  Training crawlers blocked, AI search and assistant crawlers allowed
• Blocks some AI search 1,074 (10.7%)
  At least one AI search or assistant crawler blocked
• Blocks all AI search 132 (1.3%)
  Every AI search and assistant crawler blocked
• No robots.txt 1,086 (10.9%)
  No robots.txt file: open to every crawler
• Unreadable 1,683 (16.8%)
  robots.txt could not be read from our datacenter IP (403, bot wall, connection error)

Changes since yesterday

364 changes on 77 sites since 2026-11-13: 142 crawler rules newly blocked, 121 unblocked.

Showing the first 200. The JSON download has the full list.

Block rate over time

The top 100 sites

✕ blocked · ◐ partially blocked (some paths) · ✓ allowed · ? robots.txt unreadable. "#" is the position in our sample, ordered by popularity.

Browse the index

Every site has its own page with all 29 crawlers, llms.txt and Content-Signal: all 10,000 sites. Every crawler has a page listing the sites that block it: all 29 AI crawlers.

Popular questions: Which websites block GPTBot? · Which websites block ClaudeBot? · Which websites block PerplexityBot? · Which websites block Google-Extended? · Which websites block CCBot? · Which websites block OAI-SearchBot?

By category: AI · E-commerce · Education · Entertainment · Finance · Government · News & media · Other · Reference · Social · Tech · Telecom · Travel

Download the data

• sites.csv one row per site, one column per crawler (29)
• data.json everything on this page: per-crawler and per-category stats, changes, trend and sites

Free to reuse under CC BY 4.0. Please cite: TidyTools AI Crawler Index, 2026-11-14, https://tools.yukai.uk/ai-crawler-index

Methodology & caveats

• Sites. 10,000 domains: the most popular sites of the Tranco list (list ID Y83YG) with CDN, ad, tracking, adult and gambling domains and duplicate brand domains removed, plus a few sites from an earlier study. Tranco: Le Pochat et al., Tranco: A Research-Oriented Top Sites Ranking Hardened Against Manipulation, NDSS 2019, https://doi.org/10.14722/ndss.2019.23386. We use the domain names as a sample only; the Tranco list itself and exact ranks are not redistributed.
• robots.txt only. We read https://<site>/robots.txt once a day and evaluate the rules for the home page (/) with standard robots.txt matching (RFC 9309). Firewalls, CDN bot rules, paywalls and licensing deals are not visible here: a site can allow a crawler in robots.txt and still block it at the network level, or the other way round.
• Datacenter IP, honest user agent. Requests come from one US datacenter IP with the user agent TidyToolsAICrawlerIndex/1.0 (+https://apify.com/tidytools/ai-crawler-access-checker). 1,683 sites (16.8%) could not be read: 1,037 answered 401/403, the rest had a bot wall or a connection, DNS or TLS error. They count as unknown and are excluded from the percentages.
• Blocked, partial, allowed. Blocked = the home page is disallowed for that crawler. Partial = the home page is allowed but some paths are disallowed. A crawler a site does not name follows the * group. 1,086 sites have no robots.txt at all and count as allowing everything.
• Crawlers. 29 documented AI user agents (list version 29-19a6fb23, updated 2026-10-01), the same logic as the AI Crawler Access Checker.
• llms.txt counts when /llms.txt returns non-empty plain text (not an HTML page). Content-Signal is Cloudflare's robots.txt extension (search, ai-input, ai-train).
• Changes compare today with the previous run; a site whose robots.txt is temporarily unreadable keeps its last known rules for the comparison.