TidyToolsAI Crawler Index · data as of · updated daily

AI Crawler Index · category

Do e-commerce websites block AI crawlers?

As of 2026-10-01, 10.3% of the 58 e-commerce websites whose robots.txt we could read block GPTBot, 12.1% block ClaudeBot and 6.9% block PerplexityBot (all sites: 17%, 16.3%, 12.7%). 10 of 73 publish an llms.txt file.

Block rate by crawler: e-commerce vs all sites

CrawlerOperatorE-commerceAll sites
BytespiderByteDance15.5%17.7%
CCBotCommon Crawl13.8%19.5%
ClaudeBotAnthropic12.1%16.3%
GPTBotOpenAI10.3%17%
Meta-ExternalAgentMeta10.3%14.4%
AmazonbotAmazon10.3%13.1%
Google-ExtendedGoogle8.6%14%
PerplexityBotPerplexity6.9%12.7%
Applebot-ExtendedApple6.9%14.3%
OAI-AdsBotOpenAI5.2%4.3%
Claude-SearchBotAnthropic5.2%8.1%
Claude-UserAnthropic5.2%8.3%
Perplexity-UserPerplexity5.2%8.9%
meta-webindexerMeta5.2%6.3%
meta-externalfetcherMeta5.2%8.7%
DuckAssistBotDuckDuckGo5.2%8.1%
MistralAI-UserMistral5.2%8.4%
cohere-aiCohere5.2%13.7%
OAI-SearchBotOpenAI3.4%6.7%
ChatGPT-UserOpenAI3.4%9.6%
Google-CloudVertexBotGoogle3.4%7.2%
Google-GeminiNotebookGoogle3.4%4.3%
Google-AgentGoogle3.4%4.5%
ApplebotApple3.4%2.6%
meta-externaladsMeta3.4%3.8%
Amzn-SearchBotAmazon3.4%5.9%
Amzn-UserAmazon3.4%5.5%
MistralAI-IndexMistral3.4%4.7%
MistralAI-TrainingMistral3.4%4.7%

All 73 e-commerce sites in the index

✕ blocked · ◐ partially blocked (some paths) · ✓ allowed · ? robots.txt unreadable. "#" is the position in our popularity-ordered sample.

#SitePolicyGPTBotOAI-SearchChatGPT-UserClaudeBotClaude-SearchPerplexityGoogle-Ext.CCBotllms.txt
12amazon.comBlocks some AI searchblockedblockedblockedblockedblockedblockedblockedblocked–
71shopify.comOpenallowedallowedallowedallowedallowedallowedallowedallowedyes
86ebay.comBlocks some AI searchblockedpartialpartialblockedpartialblockedallowedblocked–
98ozon.ruUnreadableunknownunknownunknownunknownunknownunknownunknownunknown–
129alibaba.comBlocks training onlyblockedallowedallowedallowedallowedallowedblockedallowed–
139etsy.comOpenallowedallowedallowedallowedallowedallowedallowedallowedyes
179aliexpress.comOpenallowedallowedallowedallowedallowedallowedallowedallowed–
186temu.comBlocks training onlypartialpartialpartialpartialpartialpartialpartialallowed–
189taobao.comOpenallowedallowedallowedallowedallowedallowedallowedallowed–
194walmart.comOpenallowedallowedallowedallowedallowedallowedallowedallowed–
215shein.comOpenallowedallowedallowedallowedallowedallowedallowedallowedyes
225wildberries.ruNo robots.txtallowedallowedallowedallowedallowedallowedallowedallowed–
244rakuten.co.jpOpenallowedallowedallowedallowedallowedallowedallowedallowed–
251shop.appOpenallowedallowedallowedallowedallowedallowedallowedallowedyes
260target.comOpenallowedallowedallowedallowedallowedallowedallowedallowedyes
264avito.ruOpenallowedallowedallowedallowedallowedallowedallowedallowed–
276ikea.comOpenallowedallowedallowedallowedallowedallowedallowedallowed–
277nike.comOpenallowedallowedallowedallowedallowedallowedallowedallowed–
287shopee.com.brOpenallowedallowedallowedallowedallowedallowedallowedallowed–
297allegro.plOpenallowedallowedallowedallowedallowedallowedallowedallowedyes
327bandcamp.comBlocks training onlyblockedallowedallowedblockedallowedallowedblockedblocked–
334homedepot.comOpenallowedallowedallowedallowedallowedallowedallowedallowed–
339zillow.comBlocks training onlyallowedallowedallowedallowedallowedallowedallowedallowed–
343bestbuy.comOpenpartialpartialpartialpartialallowedpartialpartialpartial–
347craigslist.orgOpenallowedallowedallowedallowedallowedallowedallowedallowed–
350fiverr.comOpenallowedallowedallowedallowedallowedallowedallowedallowedyes
375wb.ruNo robots.txtallowedallowedallowedallowedallowedallowedallowedallowed–
389kleinanzeigen.deUnreadableunknownunknownunknownunknownunknownunknownunknownunknown–
404coupang.comNo robots.txtallowedallowedallowedallowedallowedallowedallowedallowed–
412mercadolivre.com.brUnreadableunknownunknownunknownunknownunknownunknownunknownunknown–
415mercadolibre.com.arUnreadableunknownunknownunknownunknownunknownunknownunknownunknown–
418lowes.comUnreadableunknownunknownunknownunknownunknownunknownunknownunknown–
424jd.comOpenallowedallowedallowedallowedallowedallowedallowedallowed–
428otto.deOpenpartialpartialpartialallowedallowedpartialallowedallowed–
443flipkart.comUnreadableunknownunknownunknownunknownunknownunknownunknownunknown–
444costco.comUnreadableunknownunknownunknownunknownunknownunknownunknownunknown–
462kohls.comUnreadableunknownunknownunknownunknownunknownunknownunknownunknown–
464olx.com.brUnreadableunknownunknownunknownunknownunknownunknownunknownunknown–
469themeforest.netBlocks training onlyallowedallowedallowedallowedallowedallowedallowedblocked–
481skroutz.grBlocks training onlypartialpartialpartialpartialpartialpartialpartialblocked–
488redfin.comOpenallowedallowedallowedallowedallowedallowedallowedallowed–
501finn.noOpenpartialpartialpartialpartialallowedallowedpartialpartial–
543indiamart.comBlocks training onlyallowedallowedallowedblockedallowedallowedallowedallowed–
546leboncoin.frBlocks training onlypartialpartialpartialpartialpartialpartialpartialpartial–
547sahibinden.comUnreadableunknownunknownunknownunknownunknownunknownunknownunknown–
559rightmove.co.ukBlocks training onlyblockedallowedallowedallowedallowedallowedallowedblocked–
561meesho.comUnreadableunknownunknownunknownunknownunknownunknownunknownunknown–
563marktplaats.nlOpenallowedallowedallowedallowedallowedallowedallowedallowedyes
565tori.fiOpenallowedallowedallowedallowedallowedallowedallowedallowed–
568digikala.comOpenallowedallowedallowedallowedallowedallowedallowedallowed–
574divar.irOpenallowedallowedallowedallowedallowedallowedallowedallowed–
575idealista.comOpenallowedallowedallowedallowedallowedallowedallowedallowed–
581argos.co.ukUnreadableunknownunknownunknownunknownunknownunknownunknownunknown–
585emag.roBlocks all AI searchblockedblockedblockedblockedblockedblockedblockedblocked–
599cdiscount.comUnreadableunknownunknownunknownunknownunknownunknownunknownunknown–
615justdial.comBlocks training onlyallowedallowedallowedblockedallowedallowedallowedallowed–
620kaspi.kzOpenallowedallowedallowedallowedallowedallowedallowedallowed–
622suumo.jpOpenallowedallowedallowedallowedallowedallowedallowedallowed–
627subito.itUnreadableunknownunknownunknownunknownunknownunknownunknownunknown–
632mediaexpert.plOpenallowedallowedallowedallowedallowedallowedallowedallowed–
645leroymerlin.frOpenallowedallowedallowedallowedallowedallowedallowedallowed–
668carsensor.netOpenallowedallowedallowedallowedallowedallowedallowedallowed–
670property24.comOpenallowedallowedallowedallowedallowedallowedallowedallowed–
677action.comOpenallowedallowedallowedallowedallowedallowedallowedallowed–
679otomoto.plOpenallowedallowedallowedallowedallowedallowedallowedallowed–
691cardmarket.comUnreadableunknownunknownunknownunknownunknownunknownunknownunknown–
705markt.deOpenallowedallowedallowedallowedallowedallowedallowedallowedyes
713torob.comOpenallowedallowedallowedallowedallowedallowedallowedallowed–
788googll.storeOpenallowedallowedallowedallowedallowedallowedallowedallowed–
882marketwatch.comBlocks some AI searchpartialpartialpartialblockedblockedblockedblockedblocked–
930disdarrummers.shopBlocks some AI searchallowedallowedallowedallowedallowedallowedallowedallowed–
1004zalando.comNo robots.txtallowedallowedallowedallowedallowedallowedallowedallowed–
1005allbirds.comOpenallowedallowedallowedallowedallowedallowedallowedallowedyes

FAQ

Do e-commerce websites block AI crawlers?
As of 2026-10-01, 10.3% of the 58 e-commerce websites whose robots.txt we could read block GPTBot, 12.1% block ClaudeBot and 6.9% block PerplexityBot (all sites: 17%, 16.3%, 12.7%). 10 of 73 publish an llms.txt file.
Which e-commerce websites block GPTBot?
amazon.com, ebay.com, alibaba.com, bandcamp.com, rightmove.co.uk, emag.ro.

Other categories