TidyToolsAI Crawler Index · data as of · updated daily

AI Crawler Index · country

Do websites in Germany (.de) block AI crawlers?

As of 2026-10-03, 23.6% of the 203 websites in Germany (.de) whose robots.txt we could read block GPTBot, 20.2% block ClaudeBot and 13.3% block PerplexityBot (all 8,357 sites: 12%, 11%, 8.3%). 16 of them (7.9%) serve an llms.txt (all sites: 15.2%).

23.6%
block GPTBot (all sites 12%)
20.2%
block ClaudeBot (all sites 11%)
7.9%
serve an llms.txt (all sites 15.2%)
203
of 224 sites with a readable robots.txt

Block rate by crawler: .de (Germany) vs all sites

CrawlerOperator.de (Germany)All sitesDifference
CCBotCommon Crawl23.6%12.8%+10.8 pp
GPTBotOpenAI23.6%12%+11.6 pp
BytespiderByteDance22.2%12.3%+9.9 pp
Applebot-ExtendedApple21.7%9.6%+12.1 pp
ClaudeBotAnthropic20.2%11%+9.2 pp
Google-ExtendedGoogle20.2%9.5%+10.7 pp
Meta-ExternalAgentMeta17.2%9.5%+7.7 pp
cohere-aiCohere13.3%9.1%+4.2 pp
PerplexityBotPerplexity13.3%8.3%+5 pp
AmazonbotAmazon12.3%9.2%+3.1 pp
ChatGPT-UserOpenAI8.4%7.5%+0.9 pp
MistralAI-UserMistral7.9%5.4%+2.5 pp
DuckAssistBotDuckDuckGo7.4%5.8%+1.6 pp
OAI-SearchBotOpenAI6.4%5.7%+0.7 pp
Claude-SearchBotAnthropic5.9%5.4%+0.5 pp
Perplexity-UserPerplexity5.9%5.7%+0.2 pp
meta-externalfetcherMeta5.4%6%-0.6 pp
meta-webindexerMeta5.4%4.6%+0.8 pp
Claude-UserAnthropic4.9%5.5%-0.6 pp
Google-CloudVertexBotGoogle4.9%5.2%-0.3 pp
Amzn-SearchBotAmazon3.9%4.1%-0.2 pp
Amzn-UserAmazon3.9%3.9%0 pp
Google-AgentGoogle3.9%3.2%+0.7 pp
Google-GeminiNotebookGoogle3.4%3%+0.4 pp
meta-externaladsMeta3.4%2.8%+0.6 pp
MistralAI-IndexMistral3.4%3.3%+0.1 pp
MistralAI-TrainingMistral3.4%3.3%+0.1 pp
OAI-AdsBotOpenAI3.4%3%+0.4 pp
ApplebotApple2.5%3.1%-0.6 pp

Who blocks what, by name: GPTBot, ClaudeBot, PerplexityBot, Google-Extended, CCBot, Bytespider, Applebot-Extended, OAI-SearchBot.

Policy mix

PolicySitesShare
Open12957.6%
Blocks some AI search3415.2%
Unreadable219.4%
Blocks training only198.5%
No robots.txt198.5%
Blocks all AI search20.9%

Most restrictive .de (Germany) sites

  1. arcor.de 29 of 29 blocked
  2. qrco.de 29 of 29 blocked
  3. dwd.de 28 of 29 blocked
  4. merkur.de 21 of 29 blocked
  5. fr.de 21 of 29 blocked
  6. ruhr24.de 21 of 29 blocked
  7. hna.de 21 of 29 blocked
  8. augsburger-allgemeine.de 19 of 29 blocked
  9. derwesten.de 19 of 29 blocked
  10. rp-online.de 17 of 29 blocked
  11. mopo.de 17 of 29 blocked
  12. gala.de 16 of 29 blocked
  13. shz.de 16 of 29 blocked
  14. posteo.de 15 of 29 blocked
  15. spiegel.de 13 of 29 blocked

Sites with an llms.txt (16)

Top 100 of 224 .de (Germany) sites

✕ blocked · ◐ partially blocked (some paths) · ✓ allowed · ? robots.txt unreadable. "#" is the position in our popularity-ordered sample.

#SitePolicyGPTBotOAI-SearchChatGPT-UserClaudeBotClaude-SearchPerplexityGoogle-Ext.CCBotllms.txt
90t-online.deBlocks some AI searchblockedallowedallowedblockedallowedblockedpartialblocked–
119mobile.deOpenallowedallowedallowedallowedallowedallowedallowedallowed–
342bild.deBlocks some AI searchallowedallowedallowedallowedallowedblockedallowedblocked–
357welt.deBlocks some AI searchallowedallowedallowedallowedallowedblockedallowedblocked–
394focus.deUnreadableunknownunknownunknownunknownunknownunknownunknownunknown–
395kleinanzeigen.deUnreadableunknownunknownunknownunknownunknownunknownunknownunknown–
431otto.deOpenpartialpartialpartialallowedallowedpartialallowedallowed–
468bund.deOpenallowedallowedallowedallowedallowedallowedallowedallowed–
478n-tv.deBlocks training onlyblockedallowedallowedallowedallowedallowedblockedblocked–
516wetteronline.deNo robots.txtallowedallowedallowedallowedallowedallowedallowedallowed–
613spiegel.deBlocks some AI searchblockedblockedblockedblockedblockedallowedallowedblocked–
716markt.deOpenallowedallowedallowedallowedallowedallowedallowedallowedyes
881merkur.deBlocks some AI searchblockedpartialpartialblockedpartialpartialblockedblocked–
1156ionos.deOpenallowedallowedallowedallowedallowedallowedallowedallowed–
1166webgo.deNo robots.txtallowedallowedallowedallowedallowedallowedallowedallowed–
1225mpg.deOpenallowedallowedallowedallowedallowedallowedallowedallowed–
1250admatic.deOpenallowedallowedallowedallowedallowedallowedallowedallowed–
1270strato.deOpenallowedallowedallowedallowedallowedallowedallowedallowed–
1300ioam.deNo robots.txtallowedallowedallowedallowedallowedallowedallowedallowed–
1336zeit.deBlocks some AI searchblockedallowedallowedblockedallowedblockedblockedblocked–
1355ccc.deOpenallowedallowedallowedallowedallowedallowedallowedallowed–
1397heise.deNo robots.txtallowedallowedallowedallowedallowedallowedallowedallowed–
1484sueddeutsche.deBlocks some AI searchblockedallowedblockedblockedallowedblockedblockedallowed–
1541check24.deOpenallowedallowedallowedallowedallowedallowedallowedallowed–
1564freenet.deOpenallowedallowedallowedallowedallowedallowedallowedallowed–
1586elster.deNo robots.txtallowedallowedallowedallowedallowedallowedallowedallowed–
1620joyn.deOpenallowedallowedallowedallowedallowedallowedallowedallowed–
1635chip.deUnreadableunknownunknownunknownunknownunknownunknownunknownunknown–
1641idealo.deUnreadableunknownunknownunknownunknownunknownunknownunknownunknown–
1658tagesschau.deBlocks training onlyblockedallowedallowedblockedallowedallowedblockedblocked–
1789zdf.deOpenpartialallowedpartialallowedallowedallowedpartialpartial–
1808tagesspiegel.deBlocks training onlyblockedallowedallowedblockedallowedallowedpartialblocked–
1821e-recht24.deOpenallowedallowedallowedallowedallowedallowedallowedallowedyes
1843your-storageshare.deNo robots.txtallowedallowedallowedallowedallowedallowedallowedallowed–
1849bahn.deOpenallowedallowedallowedallowedallowedallowedallowedallowed–
1902gesetze-im-internet.deUnreadableunknownunknownunknownunknownunknownunknownunknownunknown–
1907ndr.deBlocks training onlyblockedallowedallowedblockedallowedallowedblockedblocked–
1927fraunhofer.deOpenallowedallowedallowedallowedallowedallowedallowedallowed–
1937br.deBlocks training onlyblockedallowedallowedblockedallowedallowedblockedblocked–
1950wdr.deBlocks training onlyblockedallowedallowedblockedallowedallowedblockedblocked–
1993adscale.deNo robots.txtallowedallowedallowedallowedallowedallowedallowedallowed–
2000ardmediathek.deBlocks training onlyblockedallowedallowedblockedallowedallowedblockedblocked–
2005arbeitsagentur.deOpenallowedallowedallowedallowedallowedallowedallowedallowed–
2017immobilienscout24.deOpenpartialallowedallowedpartialallowedallowedallowedallowed–
2049telefonica.deOpenallowedallowedallowedallowedallowedallowedallowedallowed–
2152stern.deBlocks some AI searchblockedallowedblockedallowedallowedallowedblockedblocked–
2153bayern.deOpenallowedallowedallowedallowedallowedallowedallowedallowed–
2159tum.deOpenallowedallowedallowedallowedallowedallowedallowedallowedyes
2182rwth-aachen.deOpenallowedallowedallowedallowedallowedallowedallowedallowed–
22151und1.deOpenallowedallowedallowedallowedallowedallowedallowedallowedyes
2221goethe.deBlocks training onlyallowedallowedallowedallowedallowedallowedallowedallowed–
2298kicker.deBlocks some AI searchblockedallowedblockedblockedblockedblockedblockedblocked–
2312swr.deBlocks training onlyblockedallowedallowedblockedallowedallowedblockedblocked–
2447uni-heidelberg.deOpenallowedallowedallowedallowedallowedallowedallowedallowed–
2459mdr.deBlocks training onlyblockedallowedallowedblockedallowedallowedblockedblocked–
2497hessen.deOpenallowedallowedallowedallowedallowedallowedallowedallowed–
2538dfn.deOpenallowedallowedallowedallowedallowedallowedallowedallowed–
2558rtl.deBlocks training onlyblockedallowedallowedallowedallowedallowedblockedblocked–
2586hosteurope.deOpenallowedallowedallowedallowedallowedallowedallowedallowed–
2651fu-berlin.deOpenallowedallowedallowedallowedallowedallowedallowedallowed–
2787kaufland.deUnreadableunknownunknownunknownunknownunknownunknownunknownunknown–
2864volkswagen.deOpenallowedallowedallowedallowedallowedallowedallowedallowed–
2940berlin.deOpenallowedallowedallowedallowedallowedallowedallowedallowed–
2999goneo.deOpenallowedallowedallowedallowedallowedallowedallowedallowed–
3023fr.deBlocks some AI searchblockedpartialpartialblockedpartialpartialblockedblocked–
3066uni-erlangen.deOpenallowedallowedallowedallowedallowedallowedallowedallowed–
3093thalia.deUnreadableunknownunknownunknownunknownunknownunknownunknownunknown–
3117vodafone.deOpenallowedallowedallowedallowedallowedallowedallowedallowed–
3180adac.deOpenallowedallowedallowedallowedallowedallowedallowedallowed–
3205hu-berlin.deOpenallowedallowedallowedallowedallowedallowedallowedallowed–
3242immowelt.deOpenpartialpartialpartialpartialpartialpartialallowedallowedyes
3283uni-hamburg.deOpenallowedallowedallowedallowedallowedallowedallowedallowed–
3285autoscout24.deBlocks training onlyblockedallowedallowedblockedallowedallowedblockedblocked–
3310uni-muenchen.deUnreadableunknownunknownunknownunknownunknownunknownunknownunknown–
3337mediamarkt.deUnreadableunknownunknownunknownunknownunknownunknownunknownunknown–
3363bfv.deNo robots.txtallowedallowedallowedallowedallowedallowedallowedallowed–
3431uni-bonn.deOpenallowedallowedallowedallowedallowedallowedallowedallowed–
3469netcologne.deOpenallowedallowedallowedallowedallowedallowedallowedallowed–
3504giga.deBlocks some AI searchblockedallowedallowedblockedallowedblockedblockedblocked–
3536uni-koeln.deOpenpartialallowedallowedallowedallowedallowedallowedallowed–
3604uni-frankfurt.deBlocks training onlyallowedallowedallowedallowedallowedallowedallowedblocked–
3633bundestag.deOpenallowedallowedallowedallowedallowedallowedallowedallowed–
3694uni-tuebingen.deOpenallowedallowedallowedallowedallowedallowedallowedallowed–
3695juris.deOpenallowedallowedallowedallowedallowedallowedallowedallowed–
3720sport1.deOpenallowedallowedallowedallowedallowedallowedallowedallowed–
3746dm.deOpenallowedallowedallowedallowedallowedallowedallowedallowedyes
3901uni-stuttgart.deOpenallowedallowedallowedallowedallowedallowedallowedallowed–
4025sportschau.deBlocks training onlyblockedallowedallowedblockedallowedallowedblockedblocked–
4042rp-online.deBlocks some AI searchblockedblockedblockedblockedblockedblockedblockedblocked–
4129niedersachsen.deOpenallowedallowedallowedallowedallowedallowedallowedallowed–
4133tvspielfilm.deUnreadableunknownunknownunknownunknownunknownunknownunknownunknown–
4155schwaebische.deBlocks some AI searchblockedblockedblockedblockedallowedblockedblockedblocked–
4157tu-dresden.deOpenallowedallowedallowedallowedallowedallowedallowedallowed–
4196tchibo.deOpenallowedallowedallowedallowedallowedallowedallowedallowedyes
4215comdirect.deOpenallowedallowedallowedallowedallowedallowedallowedallowed–
4288gala.deBlocks some AI searchblockedallowedallowedblockedblockedblockedblockedblocked–
4345hornbach.deNo robots.txtallowedallowedallowedallowedallowedallowedallowedallowed–
4348uni-goettingen.deOpenallowedallowedallowedallowedallowedallowedallowedallowed–
4359nordkurier.deBlocks some AI searchblockedblockedblockedblockedallowedblockedblockedblocked–
4379bundesregierung.deOpenallowedallowedallowedallowedallowedallowedallowedallowed–

Sorted by popularity; the rest are on the A-Z lists.

Categories on .de (Germany)

FAQ

Do websites in Germany (.de) block AI crawlers?
As of 2026-10-03, 23.6% of the 203 websites in Germany (.de) whose robots.txt we could read block GPTBot, 20.2% block ClaudeBot and 13.3% block PerplexityBot (all 8,357 sites: 12%, 11%, 8.3%). 16 of them (7.9%) serve an llms.txt (all sites: 15.2%).
Which .de (Germany) websites block the most AI crawlers?
arcor.de (29 of 29), qrco.de (29 of 29), dwd.de (28 of 29), merkur.de (21 of 29), fr.de (21 of 29). The full per-site status is on each site page.
How is this measured?
We fetch each site's robots.txt every day from one residential IP in Taiwan with an honest user agent, and apply the rules to 29 AI crawlers for the home page. 224 .de (Germany) sites are in the sample, 203 had a readable robots.txt (21 did not answer or returned an error page).

Other countries and suffixes

Compare all suffixes