TidyToolsAI Crawler Index · data as of · updated daily

AI Crawler Index · country

Do websites in United Kingdom (.uk) block AI crawlers?

As of 2026-10-03, 20% of the 90 websites in United Kingdom (.uk) whose robots.txt we could read block GPTBot, 21.1% block ClaudeBot and 21.1% block PerplexityBot (all 8,357 sites: 12%, 11%, 8.3%). 8 of them (8.9%) serve an llms.txt (all sites: 15.2%).

20%
block GPTBot (all sites 12%)
21.1%
block ClaudeBot (all sites 11%)
8.9%
serve an llms.txt (all sites 15.2%)
90
of 116 sites with a readable robots.txt

Block rate by crawler: .uk (United Kingdom) vs all sites

CrawlerOperator.uk (United Kingdom)All sitesDifference
CCBotCommon Crawl24.4%12.8%+11.6 pp
ClaudeBotAnthropic21.1%11%+10.1 pp
PerplexityBotPerplexity21.1%8.3%+12.8 pp
Applebot-ExtendedApple20%9.6%+10.4 pp
GPTBotOpenAI20%12%+8 pp
Meta-ExternalAgentMeta20%9.5%+10.5 pp
OAI-SearchBotOpenAI16.7%5.7%+11 pp
BytespiderByteDance13.3%12.3%+1 pp
AmazonbotAmazon12.2%9.2%+3 pp
Claude-SearchBotAnthropic11.1%5.4%+5.7 pp
Claude-UserAnthropic11.1%5.5%+5.6 pp
cohere-aiCohere11.1%9.1%+2 pp
Google-CloudVertexBotGoogle11.1%5.2%+5.9 pp
DuckAssistBotDuckDuckGo10%5.8%+4.2 pp
Google-ExtendedGoogle10%9.5%+0.5 pp
meta-externalfetcherMeta10%6%+4 pp
MistralAI-UserMistral10%5.4%+4.6 pp
Perplexity-UserPerplexity10%5.7%+4.3 pp
ChatGPT-UserOpenAI8.9%7.5%+1.4 pp
Amzn-SearchBotAmazon5.6%4.1%+1.5 pp
meta-webindexerMeta5.6%4.6%+1 pp
Amzn-UserAmazon4.4%3.9%+0.5 pp
MistralAI-IndexMistral4.4%3.3%+1.1 pp
OAI-AdsBotOpenAI4.4%3%+1.4 pp
Google-AgentGoogle3.3%3.2%+0.1 pp
Google-GeminiNotebookGoogle3.3%3%+0.3 pp
meta-externaladsMeta3.3%2.8%+0.5 pp
MistralAI-TrainingMistral3.3%3.3%0 pp
ApplebotApple2.2%3.1%-0.9 pp

Who blocks what, by name: GPTBot, ClaudeBot, PerplexityBot, Google-Extended, CCBot, Bytespider, Applebot-Extended, OAI-SearchBot.

Policy mix

PolicySitesShare
Open5850%
Unreadable2622.4%
Blocks some AI search1916.4%
No robots.txt86.9%
Blocks training only43.4%
Blocks all AI search10.9%

Most restrictive .uk (United Kingdom) sites

  1. england.nhs.uk 29 of 29 blocked
  2. thesun.co.uk 25 of 29 blocked
  3. thetimes.co.uk 25 of 29 blocked
  4. belfasttelegraph.co.uk 20 of 29 blocked
  5. metro.co.uk 19 of 29 blocked
  6. inews.co.uk 19 of 29 blocked
  7. telegraph.co.uk 18 of 29 blocked
  8. oxfordmail.co.uk 17 of 29 blocked
  9. gq-magazine.co.uk 17 of 29 blocked
  10. nottingham.ac.uk 16 of 29 blocked
  11. guardian.co.uk 12 of 29 blocked
  12. mirror.co.uk 7 of 29 blocked
  13. manchestereveningnews.co.uk 7 of 29 blocked
  14. walesonline.co.uk 7 of 29 blocked
  15. liverpoolecho.co.uk 7 of 29 blocked

Sites with an llms.txt (8)

Top 100 of 116 .uk (United Kingdom) sites

✕ blocked · ◐ partially blocked (some paths) · ✓ allowed · ? robots.txt unreadable. "#" is the position in our popularity-ordered sample.

#SitePolicyGPTBotOAI-SearchChatGPT-UserClaudeBotClaude-SearchPerplexityGoogle-Ext.CCBotllms.txt
127www.gov.ukOpenallowedallowedallowedallowedallowedallowedallowedallowed–
276telegraph.co.ukBlocks some AI searchblockedblockedblockedblockedblockedblockedallowedblocked–
292service.gov.ukNo robots.txtallowedallowedallowedallowedallowedallowedallowedallowed–
327independent.co.ukOpenallowedallowedallowedallowedallowedallowedallowedallowed–
458ox.ac.ukUnreadableunknownunknownunknownunknownunknownunknownunknownunknown–
567rightmove.co.ukBlocks training onlyblockedallowedallowedallowedallowedallowedallowedblocked–
594argos.co.ukUnreadableunknownunknownunknownunknownunknownunknownunknownunknown–
731cam.ac.ukOpenallowedallowedallowedallowedallowedallowedallowedallowed–
831ico.org.ukOpenallowedallowedallowedallowedallowedallowedallowedallowed–
901autotrader.co.ukUnreadableunknownunknownunknownunknownunknownunknownunknownunknown–
930www.nhs.ukOpenallowedallowedallowedallowedallowedallowedallowedallowed–
1080thesun.co.ukBlocks some AI searchpartialpartialpartialblockedblockedblockedblockedblocked–
1129mirror.co.ukBlocks some AI searchblockedblockedallowedblockedallowedblockedallowedblocked–
1252guardian.co.ukBlocks some AI searchallowedallowedallowedblockedblockedblockedallowedblocked–
1324express.co.ukBlocks some AI searchblockedblockedallowedblockedallowedblockedallowedblocked–
1348ucl.ac.ukUnreadableunknownunknownunknownunknownunknownunknownunknownunknown–
1589thetimes.co.ukBlocks some AI searchpartialpartialpartialblockedblockedblockedblockedblocked–
1738metro.co.ukBlocks some AI searchblockedblockedblockedblockedblockedblockedblockedblocked–
1904legislation.gov.ukOpenallowedallowedallowedallowedallowedallowedallowedallowed–
1918lse.ac.ukOpenallowedallowedallowedallowedallowedallowedallowedallowed–
1967standard.co.ukOpenallowedallowedallowedallowedallowedallowedallowedallowed–
2166next.co.ukUnreadableunknownunknownunknownunknownunknownunknownunknownunknown–
2235tfl.gov.ukOpenallowedallowedallowedallowedallowedallowedallowedallowed–
2435nationalarchives.gov.ukOpenallowedallowedallowedallowedallowedallowedallowedallowed–
2818manchester.ac.ukUnreadableunknownunknownunknownunknownunknownunknownunknownunknown–
2953just-eat.co.ukUnreadableunknownunknownunknownunknownunknownunknownunknownunknown–
2961gla.ac.ukOpenallowedallowedallowedallowedallowedallowedallowedallowed–
2993ons.gov.ukNo robots.txtallowedallowedallowedallowedallowedallowedallowedallowed–
3054greenend.org.ukUnreadableunknownunknownunknownunknownunknownunknownunknownunknown–
3105imperial.ac.ukOpenallowedallowedallowedallowedallowedallowedallowedallowedyes
3177warwick.ac.ukOpenallowedallowedallowedallowedallowedallowedallowedallowedyes
3207kcl.ac.ukOpenallowedallowedallowedallowedallowedallowedallowedallowed–
3286zoopla.co.ukUnreadableunknownunknownunknownunknownunknownunknownunknownunknown–
3364lloydsbank.co.ukNo robots.txtallowedallowedallowedallowedallowedallowedallowedallowed–
3471leeds.ac.ukOpenallowedallowedallowedallowedallowedallowedallowedallowed–
3518currys.co.ukUnreadableunknownunknownunknownunknownunknownunknownunknownunknown–
3768tate.org.ukOpenallowedallowedallowedallowedallowedallowedallowedallowed–
4003very.co.ukUnreadableunknownunknownunknownunknownunknownunknownunknownunknown–
4116tui.co.ukUnreadableunknownunknownunknownunknownunknownunknownunknownunknown–
4190newsnow.co.ukBlocks training onlyblockedallowedallowedallowedallowedallowedallowedallowed–
4258nice.org.ukOpenallowedallowedallowedallowedallowedallowedallowedallowed–
4269york.ac.ukOpenallowedallowedallowedallowedallowedallowedallowedallowed–
4326ncl.ac.ukOpenallowedallowedallowedallowedallowedallowedallowedallowed–
4327fca.org.ukOpenallowedallowedallowedallowedallowedallowedallowedallowed–
4457manchestereveningnews.co.ukBlocks some AI searchblockedblockedallowedblockedallowedblockedallowedblocked–
4464bl.ukOpenallowedallowedallowedallowedallowedallowedallowedallowed–
4589metoffice.gov.ukOpenallowedallowedallowedallowedallowedallowedallowedallowed–
4721therange.co.ukUnreadableunknownunknownunknownunknownunknownunknownunknownunknown–
5177walesonline.co.ukBlocks some AI searchblockedblockedallowedblockedallowedblockedallowedblocked–
5228spareroom.co.ukBlocks training onlypartialallowedallowedpartialallowedallowedallowedallowed–
5289gamblingcommission.gov.ukOpenallowedallowedallowedallowedallowedallowedallowedallowed–
5347st-andrews.ac.ukOpenallowedallowedallowedallowedallowedallowedallowedallowedyes
5374sainsburys.co.ukUnreadableunknownunknownunknownunknownunknownunknownunknownunknown–
5383nottingham.ac.ukBlocks some AI searchblockedblockedblockedblockedallowedblockedblockedblocked–
5490roman.co.ukOpenallowedallowedallowedallowedallowedallowedallowedallowed–
5505gamcare.org.ukUnreadableunknownunknownunknownunknownunknownunknownunknownunknown–
5601vivastreet.co.ukOpenallowedallowedallowedallowedallowedallowedallowedallowedyes
5800parliament.ukUnreadableunknownunknownunknownunknownunknownunknownunknownunknown–
5833soton.ac.ukUnreadableunknownunknownunknownunknownunknownunknownunknownunknown–
5904mangabuddy1.co.ukOpenallowedallowedallowedallowedallowedallowedallowedallowed–
6170pinayflix.ukOpenallowedallowedallowedallowedallowedallowedallowedallowed–
6447oxfordmail.co.ukBlocks some AI searchblockedblockedblockedblockedblockedblockedallowedblocked–
6491vam.ac.ukOpenallowedallowedallowedallowedallowedallowedallowedallowed–
6520ebi.ac.ukOpenallowedallowedallowedallowedallowedallowedallowedallowed–
6549citizensadvice.org.ukOpenallowedallowedallowedallowedallowedallowedallowedallowed–
6607inews.co.ukBlocks some AI searchblockedblockedblockedblockedblockedblockedblockedblocked–
6660nhm.ac.ukOpenallowedallowedallowedallowedallowedallowedallowedallowed–
6686nationaltrust.org.ukOpenallowedallowedallowedallowedallowedallowedallowedallowed–
6707liverpoolecho.co.ukBlocks some AI searchblockedblockedallowedblockedallowedblockedallowedblocked–
6712open.ac.ukUnreadableunknownunknownunknownunknownunknownunknownunknownunknown–
6724which.co.ukOpenallowedallowedallowedallowedallowedallowedallowedallowed–
6726dailystar.co.ukBlocks some AI searchblockedblockedallowedblockedallowedblockedallowedblocked–
6971exeter.ac.ukOpenallowedallowedallowedallowedallowedallowedallowedallowedyes
7006southampton.ac.ukOpenallowedallowedallowedallowedallowedallowedallowedallowed–
7095fasthosts.co.ukOpenallowedallowedallowedallowedallowedallowedallowedallowedyes
7128videyox.ukNo robots.txtallowedallowedallowedallowedallowedallowedallowedallowed–
7129bankofengland.co.ukUnreadableunknownunknownunknownunknownunknownunknownunknownunknown–
7254data.gov.ukOpenallowedallowedallowedallowedallowedallowedallowedallowed–
7281bath.ac.ukOpenallowedallowedallowedallowedallowedallowedallowedallowed–
7544coral.co.ukUnreadableunknownunknownunknownunknownunknownunknownunknownunknown–
7565bris.ac.ukOpenallowedallowedallowedpartialallowedallowedallowedallowed–
7588ncsc.gov.ukNo robots.txtallowedallowedallowedallowedallowedallowedallowedallowed–
7614qmul.ac.ukUnreadableunknownunknownunknownunknownunknownunknownunknownunknown–
7682dailyrecord.co.ukBlocks some AI searchblockedblockedallowedblockedallowedblockedallowedblocked–
7714sheffield.ac.ukOpenallowedallowedallowedallowedallowedallowedallowedallowed–
7717bristol.ac.ukOpenallowedallowedallowedpartialallowedallowedallowedallowed–
7752surrey.ac.ukOpenallowedallowedallowedallowedallowedallowedallowedallowed–
7872curryscloudbackup.co.ukOpenallowedallowedallowedallowedallowedallowedallowedallowed–
7911olxtoto.ukNo robots.txtallowedallowedallowedallowedallowedallowedallowedallowed–
8061ee.co.ukOpenallowedallowedallowedallowedallowedallowedallowedallowed–
8065birmingham.ac.ukOpenallowedallowedallowedallowedallowedallowedallowedallowed–
8099rhs.org.ukOpenallowedallowedallowedallowedallowedallowedallowedallowed–
8114bfi.org.ukNo robots.txtallowedallowedallowedallowedallowedallowedallowedallowed–
8297penguin.co.ukOpenallowedallowedallowedallowedallowedallowedallowedallowed–
8337mind.org.ukOpenallowedallowedallowedallowedallowedallowedallowedallowed–
8352barclays.co.ukOpenallowedallowedallowedallowedallowedallowedallowedallowed–
8402le.ac.ukUnreadableunknownunknownunknownunknownunknownunknownunknownunknown–
8414england.nhs.ukBlocks all AI searchblockedblockedblockedblockedblockedblockedblockedblockedyes
8490shef.ac.ukOpenallowedallowedallowedallowedallowedallowedallowedallowed–
8801faroutmagazine.co.ukBlocks some AI searchblockedallowedblockedallowedallowedallowedblockedblocked–

Sorted by popularity; the rest are on the A-Z lists.

Categories on .uk (United Kingdom)

FAQ

Do websites in United Kingdom (.uk) block AI crawlers?
As of 2026-10-03, 20% of the 90 websites in United Kingdom (.uk) whose robots.txt we could read block GPTBot, 21.1% block ClaudeBot and 21.1% block PerplexityBot (all 8,357 sites: 12%, 11%, 8.3%). 8 of them (8.9%) serve an llms.txt (all sites: 15.2%).
Which .uk (United Kingdom) websites block the most AI crawlers?
england.nhs.uk (29 of 29), thesun.co.uk (25 of 29), thetimes.co.uk (25 of 29), belfasttelegraph.co.uk (20 of 29), metro.co.uk (19 of 29). The full per-site status is on each site page.
How is this measured?
We fetch each site's robots.txt every day from one residential IP in Taiwan with an honest user agent, and apply the rules to 29 AI crawlers for the home page. 116 .uk (United Kingdom) sites are in the sample, 90 had a readable robots.txt (26 did not answer or returned an error page).

Other countries and suffixes

Compare all suffixes