TidyToolsAI Crawler Index · data as of · updated daily

AI Crawler Index · country

Do websites in Japan (.jp) block AI crawlers?

As of 2026-10-03, 6% of the 149 websites in Japan (.jp) whose robots.txt we could read block GPTBot, 5.4% block ClaudeBot and 4.7% block PerplexityBot (all 8,357 sites: 12%, 11%, 8.3%). 9 of them (6%) serve an llms.txt (all sites: 15.2%).

6%
block GPTBot (all sites 12%)
5.4%
block ClaudeBot (all sites 11%)
6%
serve an llms.txt (all sites 15.2%)
149
of 220 sites with a readable robots.txt

Block rate by crawler: .jp (Japan) vs all sites

CrawlerOperator.jp (Japan)All sitesDifference
Google-ExtendedGoogle6%9.5%-3.5 pp
GPTBotOpenAI6%12%-6 pp
AmazonbotAmazon5.4%9.2%-3.8 pp
BytespiderByteDance5.4%12.3%-6.9 pp
CCBotCommon Crawl5.4%12.8%-7.4 pp
ClaudeBotAnthropic5.4%11%-5.6 pp
Meta-ExternalAgentMeta5.4%9.5%-4.1 pp
Applebot-ExtendedApple4.7%9.6%-4.9 pp
ChatGPT-UserOpenAI4.7%7.5%-2.8 pp
cohere-aiCohere4.7%9.1%-4.4 pp
OAI-SearchBotOpenAI4.7%5.7%-1 pp
PerplexityBotPerplexity4.7%8.3%-3.6 pp
Claude-SearchBotAnthropic4%5.4%-1.4 pp
Claude-UserAnthropic4%5.5%-1.5 pp
meta-externalfetcherMeta4%6%-2 pp
Perplexity-UserPerplexity4%5.7%-1.7 pp
DuckAssistBotDuckDuckGo2%5.8%-3.8 pp
Google-CloudVertexBotGoogle2%5.2%-3.2 pp
ApplebotApple1.3%3.1%-1.8 pp
meta-webindexerMeta1.3%4.6%-3.3 pp
Amzn-SearchBotAmazon0.7%4.1%-3.4 pp
Amzn-UserAmazon0.7%3.9%-3.2 pp
Google-AgentGoogle0.7%3.2%-2.5 pp
Google-GeminiNotebookGoogle0.7%3%-2.3 pp
meta-externaladsMeta0.7%2.8%-2.1 pp
MistralAI-IndexMistral0.7%3.3%-2.6 pp
MistralAI-TrainingMistral0.7%3.3%-2.6 pp
MistralAI-UserMistral0.7%5.4%-4.7 pp
OAI-AdsBotOpenAI0.7%3%-2.3 pp

Who blocks what, by name: GPTBot, ClaudeBot, PerplexityBot, Google-Extended, CCBot, Bytespider, Applebot-Extended, OAI-SearchBot.

Policy mix

PolicySitesShare
Open9141.4%
Unreadable7132.3%
No robots.txt4018.2%
Blocks training only104.5%
Blocks some AI search73.2%
Blocks all AI search10.5%

Most restrictive .jp (Japan) sites

  1. carview.co.jp 29 of 29 blocked
  2. chunichi.co.jp 18 of 29 blocked
  3. nishinippon.co.jp 18 of 29 blocked
  4. yomiuri.co.jp 17 of 29 blocked
  5. nikkeibp.co.jp 16 of 29 blocked
  6. mainichi.jp 15 of 29 blocked
  7. tokyo-sports.co.jp 9 of 29 blocked
  8. qoo10.jp 3 of 29 blocked
  9. p-world.co.jp 2 of 29 blocked
  10. mamastar.jp 1 of 29 blocked
  11. ad-contents.jp 1 of 29 blocked
  12. nicovideo.jp 1 of 29 blocked
  13. jorudan.co.jp 1 of 29 blocked
  14. minkabu.jp 1 of 29 blocked
  15. tokubai.co.jp 1 of 29 blocked

Sites with an llms.txt (9)

Top 100 of 220 .jp (Japan) sites

✕ blocked · ◐ partially blocked (some paths) · ✓ allowed · ? robots.txt unreadable. "#" is the position in our popularity-ordered sample.

#SitePolicyGPTBotOAI-SearchChatGPT-UserClaudeBotClaude-SearchPerplexityGoogle-Ext.CCBotllms.txt
252rakuten.co.jpOpenallowedallowedallowedallowedallowedallowedallowedallowed–
358ameblo.jpOpenallowedallowedallowedallowedallowedallowedallowedallowed–
467docomo.ne.jpNo robots.txtallowedallowedallowedallowedallowedallowedallowedallowed–
486ameba.jpOpenallowedallowedallowedallowedallowedallowedallowedallowed–
530hotpepper.jpOpenallowedallowedallowedallowedallowedallowedallowedallowed–
542tbs.co.jpNo robots.txtallowedallowedallowedallowedallowedallowedallowedallowed–
545itmedia.co.jpNo robots.txtallowedallowedallowedallowedallowedallowedallowedallowed–
631suumo.jpOpenallowedallowedallowedallowedallowedallowedallowedallowed–
642tenki.jpOpenallowedallowedallowedallowedallowedallowedallowedallowed–
643weathernews.jpOpenpartialallowedallowedpartialallowedpartialpartialallowed–
650navitime.co.jpUnreadableunknownunknownunknownunknownunknownunknownunknownunknown–
653oricon.co.jpOpenallowedallowedallowedallowedallowedallowedallowedallowed–
665mbga.jpOpenallowedallowedallowedallowedallowedallowedallowedallowed–
695cmoa.jpUnreadableunknownunknownunknownunknownunknownunknownunknownunknown–
703gamewith.jpOpenallowedallowedallowedallowedallowedallowedallowedallowed–
745kakuyomu.jpOpenallowedallowedallowedallowedallowedallowedallowedallowed–
751rakuten-sec.co.jpOpenallowedallowedallowedallowedallowedallowedallowedallowedyes
756trilltrill.jpOpenallowedallowedallowedallowedallowedallowedallowedallowed–
770mamastar.jpBlocks training onlyallowedallowedallowedpartialallowedallowedallowedallowed–
774mechacomic.jpOpenallowedallowedallowedallowedallowedallowedallowedallowed–
784jra.go.jpUnreadableunknownunknownunknownunknownunknownunknownunknownunknown–
790alphapolis.co.jpOpenallowedallowedallowedallowedallowedallowedallowedallowedyes
804dto.jpUnreadableunknownunknownunknownunknownunknownunknownunknownunknown–
809ecnavi.jpOpenallowedallowedallowedallowedallowedallowedallowedallowed–
824site777.jpUnreadableunknownunknownunknownunknownunknownunknownunknownunknown–
834rsch.jpUnreadableunknownunknownunknownunknownunknownunknownunknownunknown–
837ad-contents.jpBlocks some AI searchallowedallowedallowedallowedallowedallowedallowedallowed–
913auone.jpNo robots.txtallowedallowedallowedallowedallowedallowedallowedallowed–
941granbluefantasy.jpNo robots.txtallowedallowedallowedallowedallowedallowedallowedallowed–
947suruga-ya.jpUnreadableunknownunknownunknownunknownunknownunknownunknownunknown–
979takarakuji-official.jpUnreadableunknownunknownunknownunknownunknownunknownunknownunknown–
1018sakura.ne.jpOpenallowedallowedallowedallowedallowedallowedallowedallowed–
1253hatena.ne.jpUnreadableunknownunknownunknownunknownunknownunknownunknownunknown–
1551cpi.ad.jpOpenallowedallowedallowedallowedallowedallowedallowedallowed–
1681nicovideo.jpBlocks training onlyblockedallowedallowedallowedallowedallowedallowedallowed–
1824adingo.jpNo robots.txtallowedallowedallowedallowedallowedallowedallowedallowed–
1863mhlw.go.jpUnreadableunknownunknownunknownunknownunknownunknownunknownunknown–
1914prtimes.jpOpenallowedallowedallowedallowedallowedallowedallowedallowed–
2058mynavi.jpOpenallowedallowedallowedallowedallowedallowedallowedallowed–
2106yomiuri.co.jpBlocks some AI searchblockedblockedblockedblockedblockedblockedblockedblocked–
2191japanpost.jpUnreadableunknownunknownunknownunknownunknownunknownunknownunknown–
2287mainichi.jpBlocks some AI searchblockedblockedblockedblockedblockedblockedblockedblocked–
2377japantimes.co.jpUnreadableunknownunknownunknownunknownunknownunknownunknownunknown–
2411u-tokyo.ac.jpUnreadableunknownunknownunknownunknownunknownunknownunknownunknown–
2482jst.go.jpUnreadableunknownunknownunknownunknownunknownunknownunknownunknown–
2533biglobe.ne.jpNo robots.txtallowedallowedallowedallowedallowedallowedallowedallowed–
2557recruit.co.jpUnreadableunknownunknownunknownunknownunknownunknownunknownunknown–
2645plala.or.jpUnreadableunknownunknownunknownunknownunknownunknownunknownunknown–
2676infoseek.co.jpNo robots.txtallowedallowedallowedallowedallowedallowedallowedallowed–
2707jal.co.jpUnreadableunknownunknownunknownunknownunknownunknownunknownunknown–
2715jma.go.jpUnreadableunknownunknownunknownunknownunknownunknownunknownunknown–
2727softbank.jpNo robots.txtallowedallowedallowedallowedallowedallowedallowedallowed–
2785ntv.co.jpOpenallowedallowedallowedallowedallowedallowedallowedallowed–
2877buyee.jpUnreadableunknownunknownunknownunknownunknownunknownunknownunknown–
2944tv-asahi.co.jpNo robots.txtallowedallowedallowedallowedallowedallowedallowedallowed–
2980excite.co.jpOpenallowedallowedallowedallowedallowedallowedallowedallowed–
3111gnavi.co.jpOpenallowedallowedallowedallowedallowedallowedallowedallowed–
3221pia.jpNo robots.txtallowedallowedallowedallowedallowedallowedallowedallowed–
3222ocn.ne.jpUnreadableunknownunknownunknownunknownunknownunknownunknownunknown–
3228meti.go.jpUnreadableunknownunknownunknownunknownunknownunknownunknownunknown–
3359kuronekoyamato.co.jpUnreadableunknownunknownunknownunknownunknownunknownunknownunknown–
3391mlit.go.jpUnreadableunknownunknownunknownunknownunknownunknownunknownunknown–
3396impress.co.jpOpenallowedallowedallowedallowedallowedallowedallowedallowed–
3408exblog.jpUnreadableunknownunknownunknownunknownunknownunknownunknownunknown–
3671mixi.jpOpenallowedallowedallowedallowedallowedallowedallowedallowed–
3846carview.co.jpBlocks all AI searchblockedblockedblockedblockedblockedblockedblockedblocked–
3851kyoto-u.ac.jpOpenallowedallowedallowedallowedallowedallowedallowedallowed–
3938diamond.jpOpenallowedallowedallowedallowedallowedallowedallowedallowed–
3989chunichi.co.jpBlocks some AI searchblockedblockedblockedblockedblockedblockedblockedblocked–
4055beforward.jpOpenallowedallowedallowedallowedallowedallowedallowedallowed–
4099sponichi.co.jpNo robots.txtallowedallowedallowedallowedallowedallowedallowedallowed–
4135jorudan.co.jpBlocks training onlyallowedallowedallowedallowedallowedallowedallowedallowed–
4149hmv.co.jpUnreadableunknownunknownunknownunknownunknownunknownunknownunknown–
4154fnn.jpNo robots.txtallowedallowedallowedallowedallowedallowedallowedallowed–
4214so-net.ne.jpOpenallowedallowedallowedallowedallowedallowedallowedallowed–
4224athome.co.jpOpenallowedallowedallowedallowedallowedallowedallowedallowed–
4261minkabu.jpBlocks training onlyblockedallowedallowedallowedallowedallowedallowedallowed–
4279mizuhobank.co.jpUnreadableunknownunknownunknownunknownunknownunknownunknownunknown–
4373nishinippon.co.jpBlocks some AI searchblockedblockedblockedblockedblockedblockedblockedblocked–
4383tokubai.co.jpBlocks training onlyallowedallowedallowedblockedallowedallowedallowedallowed–
4428kabutan.jpOpenallowedallowedallowedallowedallowedallowedallowedallowed–
4438mofa.go.jpUnreadableunknownunknownunknownunknownunknownunknownunknownunknown–
4445nii.ac.jpUnreadableunknownunknownunknownunknownunknownunknownunknownunknown–
4488uncn.jpOpenallowedallowedallowedallowedallowedallowedallowedallowed–
4498president.jpOpenallowedallowedallowedallowedallowedallowedallowedallowed–
4563jcb.co.jpUnreadableunknownunknownunknownunknownunknownunknownunknownunknown–
4572asahi-net.or.jpUnreadableunknownunknownunknownunknownunknownunknownunknownunknown–
4582daily.co.jpOpenallowedallowedallowedallowedallowedallowedallowedallowed–
4584wikiwiki.jpOpenallowedallowedallowedallowedallowedallowedallowedallowed–
4630sbisec.co.jpOpenallowedallowedallowedallowedallowedallowedallowedallowed–
4687papy.co.jpOpenallowedallowedallowedallowedallowedallowedallowedallowed–
4709fantia.jpOpenallowedallowedallowedallowedallowedallowedallowedallowed–
4828palcloset.jpOpenallowedallowedallowedallowedallowedallowedallowedallowedyes
4835qoo10.jpBlocks training onlyallowedallowedallowedallowedallowedallowedblockedblockedyes
4845mdpr.jpOpenallowedallowedallowedallowedallowedallowedallowedallowed–
4877booklive.jpOpenallowedallowedallowedallowedallowedallowedallowedallowedyes
4909p-bandai.jpNo robots.txtallowedallowedallowedallowedallowedallowedallowedallowed–
4957mufg.jpUnreadableunknownunknownunknownunknownunknownunknownunknownunknown–
4970melonbooks.co.jpNo robots.txtallowedallowedallowedallowedallowedallowedallowedallowed–
4987felissimo.co.jpOpenallowedallowedallowedallowedallowedallowedallowedallowed–

Sorted by popularity; the rest are on the A-Z lists.

Categories on .jp (Japan)

FAQ

Do websites in Japan (.jp) block AI crawlers?
As of 2026-10-03, 6% of the 149 websites in Japan (.jp) whose robots.txt we could read block GPTBot, 5.4% block ClaudeBot and 4.7% block PerplexityBot (all 8,357 sites: 12%, 11%, 8.3%). 9 of them (6%) serve an llms.txt (all sites: 15.2%).
Which .jp (Japan) websites block the most AI crawlers?
carview.co.jp (29 of 29), chunichi.co.jp (18 of 29), nishinippon.co.jp (18 of 29), yomiuri.co.jp (17 of 29), nikkeibp.co.jp (16 of 29). The full per-site status is on each site page.
How is this measured?
We fetch each site's robots.txt every day from one residential IP in Taiwan with an honest user agent, and apply the rules to 29 AI crawlers for the home page. 220 .jp (Japan) sites are in the sample, 149 had a readable robots.txt (71 did not answer or returned an error page).

Other countries and suffixes

Compare all suffixes