TidyToolsAI Crawler Index · data as of · updated daily

AI Crawler Index · country

Do websites in Netherlands (.nl) block AI crawlers?

As of 2026-10-03, 27.1% of the 48 websites in Netherlands (.nl) whose robots.txt we could read block GPTBot, 25% block ClaudeBot and 22.9% block PerplexityBot (all 8,357 sites: 12%, 11%, 8.3%). 2 of them (4.2%) serve an llms.txt (all sites: 15.2%).

27.1%
block GPTBot (all sites 12%)
25%
block ClaudeBot (all sites 11%)
4.2%
serve an llms.txt (all sites 15.2%)
48
of 53 sites with a readable robots.txt

Block rate by crawler: .nl (Netherlands) vs all sites

CrawlerOperator.nl (Netherlands)All sitesDifference
CCBotCommon Crawl27.1%12.8%+14.3 pp
GPTBotOpenAI27.1%12%+15.1 pp
ClaudeBotAnthropic25%11%+14 pp
BytespiderByteDance22.9%12.3%+10.6 pp
ChatGPT-UserOpenAI22.9%7.5%+15.4 pp
Google-ExtendedGoogle22.9%9.5%+13.4 pp
PerplexityBotPerplexity22.9%8.3%+14.6 pp
cohere-aiCohere20.8%9.1%+11.7 pp
OAI-SearchBotOpenAI20.8%5.7%+15.1 pp
AmazonbotAmazon18.8%9.2%+9.6 pp
Applebot-ExtendedApple18.8%9.6%+9.2 pp
Perplexity-UserPerplexity18.8%5.7%+13.1 pp
meta-externalfetcherMeta16.7%6%+10.7 pp
MistralAI-UserMistral16.7%5.4%+11.3 pp
Claude-SearchBotAnthropic8.3%5.4%+2.9 pp
DuckAssistBotDuckDuckGo8.3%5.8%+2.5 pp
Google-CloudVertexBotGoogle8.3%5.2%+3.1 pp
Meta-ExternalAgentMeta8.3%9.5%-1.2 pp
meta-webindexerMeta6.3%4.6%+1.7 pp
Claude-UserAnthropic4.2%5.5%-1.3 pp
Google-AgentGoogle4.2%3.2%+1 pp
ApplebotApple2.1%3.1%-1 pp
MistralAI-IndexMistral2.1%3.3%-1.2 pp
OAI-AdsBotOpenAI2.1%3%-0.9 pp
Amzn-SearchBotAmazon0%4.1%-4.1 pp
Amzn-UserAmazon0%3.9%-3.9 pp
Google-GeminiNotebookGoogle0%3%-3 pp
meta-externaladsMeta0%2.8%-2.8 pp
MistralAI-TrainingMistral0%3.3%-3.3 pp

Who blocks what, by name: GPTBot, ClaudeBot, PerplexityBot, Google-Extended, CCBot, Bytespider, Applebot-Extended, OAI-SearchBot.

Policy mix

PolicySitesShare
Open2750.9%
Blocks some AI search1222.6%
No robots.txt713.2%
Unreadable59.4%
Blocks training only23.8%

Most restrictive .nl (Netherlands) sites

  1. telegraaf.nl 20 of 29 blocked
  2. nrc.nl 18 of 29 blocked
  3. volkskrant.nl 18 of 29 blocked
  4. parool.nl 18 of 29 blocked
  5. ad.nl 14 of 29 blocked
  6. nu.nl 14 of 29 blocked
  7. gelderlander.nl 14 of 29 blocked
  8. destentor.nl 14 of 29 blocked
  9. nos.nl 13 of 29 blocked
  10. voetbalzone.nl 11 of 29 blocked
  11. overheid.nl 10 of 29 blocked
  12. bnr.nl 5 of 29 blocked
  13. mijnwoordenboek.nl 1 of 29 blocked
  14. autoriteitpersoonsgegevens.nl 1 of 29 blocked

Sites with an llms.txt (2)

All 53 .nl (Netherlands) sites

✕ blocked · ◐ partially blocked (some paths) · ✓ allowed · ? robots.txt unreadable. "#" is the position in our popularity-ordered sample.

#SitePolicyGPTBotOAI-SearchChatGPT-UserClaudeBotClaude-SearchPerplexityGoogle-Ext.CCBotllms.txt
540ad.nlBlocks some AI searchblockedblockedblockedblockedallowedblockedblockedblocked–
547nu.nlBlocks some AI searchblockedblockedblockedblockedallowedblockedblockedblocked–
571marktplaats.nlOpenallowedallowedallowedallowedallowedallowedallowedallowedyes
1053transip.nlOpenallowedallowedallowedallowedallowedallowedallowedallowed–
1582ziggo.nlOpenallowedallowedallowedallowedallowedallowedallowedallowed–
1897nos.nlBlocks some AI searchblockedallowedblockedblockedallowedblockedblockedblocked–
2263nlnetlabs.nlOpenallowedallowedallowedallowedallowedallowedallowedallowed–
2310cwi.nlOpenallowedallowedallowedallowedallowedallowedallowedallowed–
2456uu.nlOpenallowedallowedallowedallowedallowedallowedallowedallowed–
2633cloud86.nlOpenallowedallowedallowedallowedallowedallowedallowedallowed–
2654caiway.nlUnreadableunknownunknownunknownunknownunknownunknownunknownunknown–
2697telegraaf.nlBlocks some AI searchblockedblockedblockedblockedblockedblockedblockedblocked–
3183xs4all.nlNo robots.txtallowedallowedallowedallowedallowedallowedallowedallowed–
3259hostnet.nlOpenallowedallowedallowedallowedallowedallowedallowedallowed–
3267tudelft.nlOpenallowedallowedallowedallowedallowedallowedallowedallowed–
3400uva.nlOpenallowedallowedallowedallowedallowedallowedallowedallowed–
3449buienradar.nlOpenallowedallowedallowedallowedallowedallowedallowedallowed–
3600rug.nlOpenallowedallowedallowedallowedallowedallowedallowedallowed–
3759funda.nlNo robots.txtallowedallowedallowedallowedallowedallowedallowedallowed–
3940utwente.nlOpenallowedallowedallowedallowedallowedallowedallowedallowedyes
3946tue.nlOpenallowedallowedallowedallowedallowedallowedallowedallowed–
4125gelderlander.nlBlocks some AI searchblockedblockedblockedblockedallowedblockedblockedblocked–
4167nrc.nlBlocks some AI searchblockedblockedblockedblockedblockedblockedblockedblocked–
4252vu.nlOpenallowedallowedallowedallowedallowedallowedallowedallowed–
4285rijksoverheid.nlOpenallowedallowedallowedallowedallowedallowedallowedallowed–
4475destentor.nlBlocks some AI searchblockedblockedblockedblockedallowedblockedblockedblocked–
4718volkskrant.nlBlocks some AI searchblockedblockedblockedblockedblockedblockedblockedblocked–
5033hdfilmcehennemi.nlOpenallowedallowedallowedallowedallowedallowedallowedallowed–
5087flexwebhosting.nlOpenallowedallowedallowedallowedallowedallowedallowedallowed–
5203voetbalzone.nlBlocks some AI searchblockedblockedblockedblockedallowedblockedallowedblocked–
5238mijnwoordenboek.nlBlocks training onlyallowedallowedallowedallowedallowedallowedallowedblocked–
6108overheid.nlBlocks some AI searchblockedblockedallowedblockedallowedblockedblockedblocked–
6293racing.nlOpenallowedallowedallowedallowedallowedallowedallowedallowed–
6304autoriteitpersoonsgegevens.nlBlocks training onlyblockedallowedallowedallowedallowedallowedallowedallowed–
6646wur.nlOpenallowedallowedallowedallowedallowedallowedallowedallowed–
6662neostrada.nlOpenallowedallowedallowedallowedallowedallowedallowedallowed–
7235surfnet.nlNo robots.txtallowedallowedallowedallowedallowedallowedallowedallowed–
7336ah.nlUnreadableunknownunknownunknownunknownunknownunknownunknownunknown–
7446ru.nlNo robots.txtallowedallowedallowedallowedallowedallowedallowedallowed–
7459rabobank.nlUnreadableunknownunknownunknownunknownunknownunknownunknownunknown–
7819mijndomein.nlOpenallowedallowedallowedallowedallowedallowedallowedallowed–
7955universiteitleiden.nlOpenallowedallowedallowedallowedallowedallowedallowedallowed–
8252parool.nlBlocks some AI searchblockedblockedblockedblockedblockedblockedblockedblocked–
8314anwb.nlOpenallowedallowedallowedallowedallowedallowedallowedallowed–
8316deltafibernederland.nlNo robots.txtallowedallowedallowedallowedallowedallowedallowedallowed–
8361coolblue.nlUnreadableunknownunknownunknownunknownunknownunknownunknownunknown–
8728kpnmail.nlNo robots.txtallowedallowedallowedallowedallowedallowedallowedallowed–
8747prijsmepper.nlNo robots.txtallowedallowedallowedallowedallowedallowedallowedallowed–
8756belastingdienst.nlOpenallowedallowedallowedallowedallowedallowedallowedallowed–
9671nltimes.nlUnreadableunknownunknownunknownunknownunknownunknownunknownunknown–
9706bnr.nlBlocks some AI searchblockedallowedblockedblockedallowedallowedblockedblocked–
9778hema.nlOpenallowedallowedallowedallowedallowedallowedallowedallowed–
9947rijksmuseum.nlOpenallowedallowedallowedallowedallowedallowedallowedallowed–

Categories on .nl (Netherlands)

FAQ

Do websites in Netherlands (.nl) block AI crawlers?
As of 2026-10-03, 27.1% of the 48 websites in Netherlands (.nl) whose robots.txt we could read block GPTBot, 25% block ClaudeBot and 22.9% block PerplexityBot (all 8,357 sites: 12%, 11%, 8.3%). 2 of them (4.2%) serve an llms.txt (all sites: 15.2%).
Which .nl (Netherlands) websites block the most AI crawlers?
telegraaf.nl (20 of 29), nrc.nl (18 of 29), volkskrant.nl (18 of 29), parool.nl (18 of 29), ad.nl (14 of 29). The full per-site status is on each site page.
How is this measured?
We fetch each site's robots.txt every day from one residential IP in Taiwan with an honest user agent, and apply the rules to 29 AI crawlers for the home page. 53 .nl (Netherlands) sites are in the sample, 48 had a readable robots.txt (5 did not answer or returned an error page).

Other countries and suffixes

Compare all suffixes