TidyToolsAI Crawler Index · data as of · updated daily

AI Crawler Index · category

Do e-commerce websites block AI crawlers?

As of 2026-11-16, 14.7% of the 116 e-commerce websites whose robots.txt we could read block GPTBot, 17.2% block ClaudeBot and 5.2% block PerplexityBot (all sites: 15.4%, 14.7%, 8.2%). 18 of 139 publish an llms.txt file.

Block rate by crawler: e-commerce vs all sites

CrawlerOperatorE-commerceAll sites
CCBotCommon Crawl18.1%16.4%
BytespiderByteDance18.1%16.2%
ClaudeBotAnthropic17.2%14.7%
AmazonbotAmazon17.2%13.2%
GPTBotOpenAI14.7%15.4%
Meta-ExternalAgentMeta14.7%13.6%
Google-ExtendedGoogle13.8%13.2%
Applebot-ExtendedApple12.9%13.4%
cohere-aiCohere11.2%12.7%
MistralAI-TrainingMistral8.6%7.1%
Google-CloudVertexBotGoogle7.8%8.8%
PerplexityBotPerplexity5.2%8.2%
Claude-SearchBotAnthropic4.3%5.7%
Claude-UserAnthropic4.3%5.9%
Perplexity-UserPerplexity4.3%6.1%
OAI-SearchBotOpenAI3.4%5.8%
ChatGPT-UserOpenAI3.4%7.4%
OAI-AdsBotOpenAI3.4%3%
meta-webindexerMeta3.4%5.1%
meta-externalfetcherMeta3.4%6.5%
Amzn-SearchBotAmazon3.4%4.4%
DuckAssistBotDuckDuckGo3.4%5.9%
MistralAI-UserMistral3.4%5.7%
Google-GeminiNotebookGoogle2.6%3.3%
Google-AgentGoogle2.6%3.6%
meta-externaladsMeta2.6%2.9%
Amzn-UserAmazon2.6%4.4%
MistralAI-IndexMistral2.6%3.6%
ApplebotApple1.7%3.2%

All 139 e-commerce sites in the index

✕ blocked · ◐ partially blocked (some paths) · ✓ allowed · ? robots.txt unreadable. "#" is the position in our popularity-ordered sample.

#SitePolicyGPTBotOAI-SearchChatGPT-UserClaudeBotClaude-SearchPerplexityGoogle-Ext.CCBotllms.txt
13amazon.comBlocks some AI searchblockedblockedblockedblockedblockedblockedblockedblocked–
72shopify.comOpenallowedallowedallowedallowedallowedallowedallowedallowedyes
86ebay.comBlocks some AI searchblockedpartialpartialblockedpartialblockedallowedblocked–
101ozon.ruUnreadableunknownunknownunknownunknownunknownunknownunknownunknown–
134alibaba.comBlocks training onlyblockedallowedallowedallowedallowedallowedblockedallowed–
141etsy.comOpenallowedallowedallowedallowedallowedallowedallowedallowedyes
188aliexpress.comBlocks training onlyblockedallowedallowedblockedallowedallowedblockedblocked–
189temu.comNo robots.txtallowedallowedallowedallowedallowedallowedallowedallowed–
194taobao.comOpenallowedallowedallowedallowedallowedallowedallowedallowed–
202walmart.comOpenallowedallowedallowedallowedallowedallowedallowedallowed–
218shein.comOpenallowedallowedallowedallowedallowedallowedallowedallowedyes
235wildberries.ruNo robots.txtallowedallowedallowedallowedallowedallowedallowedallowed–
252rakuten.co.jpOpenallowedallowedallowedallowedallowedallowedallowedallowed–
259shop.appOpenallowedallowedallowedallowedallowedallowedallowedallowedyes
267target.comOpenallowedallowedallowedallowedallowedallowedallowedallowedyes
269avito.ruOpenallowedallowedallowedallowedallowedallowedallowedallowed–
280ikea.comBlocks training onlyallowedallowedallowedallowedallowedallowedallowedallowed–
282nike.comOpenallowedallowedallowedallowedallowedallowedallowedallowed–
291shopee.com.brOpenallowedallowedallowedallowedallowedallowedallowedallowed–
306allegro.plOpenallowedallowedallowedallowedallowedallowedallowedpartialyes
329bandcamp.comBlocks training onlyblockedallowedallowedblockedallowedallowedblockedblocked–
339homedepot.comUnreadableunknownunknownunknownunknownunknownunknownunknownunknown–
345zillow.comBlocks training onlyallowedallowedallowedpartialpartialallowedallowedallowed–
348bestbuy.comOpenpartialpartialpartialpartialallowedpartialpartialpartial–
351craigslist.orgOpenallowedallowedallowedallowedallowedallowedallowedallowed–
354fiverr.comOpenallowedpartialpartialallowedpartialpartialpartialallowedyes
381wb.ruNo robots.txtallowedallowedallowedallowedallowedallowedallowedallowed–
395kleinanzeigen.deUnreadableunknownunknownunknownunknownunknownunknownunknownunknown–
408coupang.comUnreadableunknownunknownunknownunknownunknownunknownunknownunknown–
416mercadolivre.com.brUnreadableunknownunknownunknownunknownunknownunknownunknownunknown–
423mercadolibre.com.arUnreadableunknownunknownunknownunknownunknownunknownunknownunknown–
425lowes.comUnreadableunknownunknownunknownunknownunknownunknownunknownunknown–
428jd.comOpenallowedallowedallowedallowedallowedallowedallowedallowed–
431otto.deOpenpartialpartialpartialallowedallowedpartialallowedallowedyes
445bol.comOpenallowedallowedallowedallowedallowedallowedallowedallowed–
447costco.comOpenallowedallowedallowedallowedallowedallowedallowedallowed–
451flipkart.comBlocks some AI searchallowedallowedallowedallowedallowedallowedallowedallowed–
469kohls.comUnreadableunknownunknownunknownunknownunknownunknownunknownunknown–
470olx.com.brUnreadableunknownunknownunknownunknownunknownunknownunknownunknown–
477themeforest.netBlocks training onlyallowedallowedallowedallowedallowedallowedallowedblocked–
494redfin.comOpenallowedallowedallowedallowedallowedallowedallowedallowed–
498skroutz.grBlocks training onlypartialpartialpartialpartialpartialpartialpartialblocked–
511finn.noOpenpartialpartialpartialpartialallowedallowedpartialpartial–
520rustore.ruOpenallowedallowedallowedallowedallowedallowedallowedallowed–
552indiamart.comBlocks training onlyallowedallowedallowedblockedallowedallowedallowedallowed–
556leboncoin.frBlocks training onlypartialpartialpartialpartialpartialpartialpartialpartial–
558sahibinden.comUnreadableunknownunknownunknownunknownunknownunknownunknownunknown–
567rightmove.co.ukBlocks training onlyblockedallowedallowedallowedallowedallowedallowedblocked–
568tori.fiOpenallowedallowedallowedallowedallowedallowedallowedallowed–
569meesho.comUnreadableunknownunknownunknownunknownunknownunknownunknownunknown–
571marktplaats.nlOpenallowedallowedallowedallowedallowedallowedallowedallowedyes
575digikala.comOpenallowedallowedallowedallowedallowedallowedallowedallowed–
581idealista.comOpenallowedallowedallowedallowedallowedallowedallowedallowed–
582divar.irOpenallowedallowedallowedallowedallowedallowedallowedallowed–
594argos.co.ukUnreadableunknownunknownunknownunknownunknownunknownunknownunknown–
597emag.roOpenallowedallowedallowedallowedallowedallowedallowedallowed–
603cdiscount.comUnreadableunknownunknownunknownunknownunknownunknownunknownunknown–
624kaspi.kzOpenallowedallowedallowedallowedallowedallowedallowedallowed–
625justdial.comBlocks training onlyallowedallowedallowedblockedallowedallowedallowedallowed–
631suumo.jpOpenallowedallowedallowedallowedallowedallowedallowedallowed–
640subito.itUnreadableunknownunknownunknownunknownunknownunknownunknownunknown–
641mediaexpert.plOpenallowedallowedallowedallowedallowedallowedallowedallowed–
656leroymerlin.frUnreadableunknownunknownunknownunknownunknownunknownunknownunknown–
676carsensor.netOpenallowedallowedallowedallowedallowedallowedallowedallowed–
678property24.comOpenallowedallowedallowedallowedallowedallowedallowedallowed–
689action.comOpenallowedallowedallowedallowedallowedallowedallowedallowed–
691otomoto.plOpenallowedallowedallowedallowedallowedallowedallowedallowed–
704cardmarket.comBlocks some AI searchblockedallowedallowedblockedallowedallowedblockedblocked–
716markt.deOpenallowedallowedallowedallowedallowedallowedallowedallowedyes
722torob.comOpenallowedallowedallowedallowedallowedallowedallowedallowed–
807googll.storeOpenallowedallowedallowedallowedallowedallowedallowedallowed–
904marketwatch.comBlocks some AI searchpartialpartialpartialblockedblockedblockedblockedblocked–
957disdarrummers.shopBlocks some AI searchallowedallowedallowedallowedallowedallowedallowedallowed–
1038videeyss.shopBlocks some AI searchblockedallowedallowedblockedallowedallowedblockedblocked–
1320tmall.comOpenallowedallowedallowedallowedallowedallowedallowedallowed–
2045smallpdf.comOpenallowedallowedallowedallowedallowedallowedallowedallowed–
2154prestashop.comOpenallowedallowedallowedallowedallowedallowedallowedallowed–
2156capitaloneshopping.comOpenallowedallowedallowedallowedallowedallowedallowedallowedyes
2231buymeacoffee.comOpenallowedallowedallowedallowedallowedallowedallowedallowed–
2238gmarket.co.krBlocks some AI searchallowedblockedblockedblockedblockedblockedblockedblocked–
2877buyee.jpUnreadableunknownunknownunknownunknownunknownunknownunknownunknown–
3106emarketer.comOpenallowedallowedallowedallowedallowedallowedpartialallowed–
3141bookshop.orgBlocks training onlyblockedpartialpartialblockedpartialpartialblockedblocked–
3300momoshop.com.twNo robots.txtallowedallowedallowedallowedallowedallowedallowedallowed–
3331onthemarket.comBlocks training onlyallowedallowedallowedblockedallowedallowedallowedallowed–
3445webstaurantstore.comUnreadableunknownunknownunknownunknownunknownunknownunknownunknown–
3502trustedshops.comBlocks training onlyallowedallowedallowedallowedallowedallowedallowedblocked–
3828comparethemarket.comUnreadableunknownunknownunknownunknownunknownunknownunknownunknown–
4130midasbuy.comBlocks all AI searchblockedblockedblockedblockedblockedblockedblockedblocked–
4393marketscreener.comOpenallowedallowedallowedallowedallowedallowedallowedallowed–
4419shop-apotheke.comOpenallowedallowedallowedallowedallowedallowedallowedallowed–
4434codashop.comOpenallowedallowedallowedallowedallowedallowedallowedallowed–
4503bigbadtoystore.comOpenallowedallowedallowedallowedallowedallowedallowedallowed–
5041shopier.comUnreadableunknownunknownunknownunknownunknownunknownunknownunknown–
5051litmarket.ruBlocks training onlyblockedallowedallowedblockedallowedallowedblockedblocked–
5217shop-pro.jpOpenallowedallowedallowedallowedallowedallowedallowedallowedyes
5428zalando.comNo robots.txtallowedallowedallowedallowedallowedallowedallowedallowed–
5595naturabuy.frOpenallowedallowedallowedallowedallowedallowedallowedallowed–
5633shopch.jpNo robots.txtallowedallowedallowedallowedallowedallowedallowedallowed–
5651shopsy.inOpenallowedallowedallowedallowedallowedallowedallowedallowed–
5684shop-shimamura.comUnreadableunknownunknownunknownunknownunknownunknownunknownunknown–
5711glorioushop.comNo robots.txtallowedallowedallowedallowedallowedallowedallowedallowed–
5994sertastore.com.brOpenallowedallowedallowedallowedallowedallowedallowedallowed–
6139myshopline.comNo robots.txtallowedallowedallowedallowedallowedallowedallowedallowed–
6179shopcider.comOpenallowedallowedallowedallowedallowedallowedallowedallowed–
6247market-qx.tradeBlocks some AI searchblockedallowedallowedblockedallowedallowedblockedblocked–
6268jilateskrimpanas.storeOpenallowedallowedallowedallowedallowedallowedallowedallowed–
6294makeshop.co.krOpenallowedallowedallowedallowedallowedallowedallowedallowed–
6360videystream.storeBlocks some AI searchblockedallowedallowedblockedallowedallowedblockedblocked–
6502rebuyengine.comOpenallowedallowedallowedallowedallowedallowedallowedallowed–
6504marketsandmarkets.comUnreadableunknownunknownunknownunknownunknownunknownunknownunknown–
6713researchandmarkets.comOpenallowedallowedallowedallowedallowedallowedallowedallowed–
6823wholefoodsmarket.comOpenallowedallowedallowedallowedallowedallowedallowedallowed–
6855yamallng.ruOpenallowedallowedallowedallowedallowedallowedallowedallowed–
6862chiikawamarket.jpOpenallowedallowedallowedallowedallowedallowedallowedallowedyes
7139mencretmesum.storeOpenallowedallowedallowedallowedallowedallowedallowedallowed–
7165letyshops.comBlocks some AI searchblockedblockedblockedblockedblockedblockedblockedblocked–
7259marketiq.comOpenallowedallowedallowedallowedallowedallowedallowedallowed–
7469360buyimg.comNo robots.txtallowedallowedallowedallowedallowedallowedallowedallowed–
7600tiktokshop.comNo robots.txtallowedallowedallowedallowedallowedallowedallowedallowed–
7735viideyss.storeBlocks some AI searchblockedallowedallowedblockedallowedallowedblockedblocked–
7811dmarket.comOpenallowedallowedallowedallowedallowedallowedallowedallowedyes
8168storeland.ruOpenallowedallowedallowedallowedallowedallowedallowedallowed–
8234moneysupermarket.comOpenpartialpartialpartialpartialpartialpartialallowedpartialyes
8236myshoptet.comUnreadableunknownunknownunknownunknownunknownunknownunknownunknown–
8259shopmy.usOpenallowedallowedallowedallowedallowedallowedallowedallowedyes
8454contentmarketinginstitute.comBlocks training onlyblockedallowedallowedblockedallowedallowedallowedblocked–
8461fakestoreapi.comOpenallowedallowedallowedallowedallowedallowedallowedallowed–
8567shopbop.comOpenallowedallowedallowedallowedallowedallowedallowedallowed–
8803marketplace.orgNo robots.txtallowedallowedallowedallowedallowedallowedallowedallowed–
8883stores.jpOpenallowedallowedallowedallowedallowedallowedallowedallowed–
8901creativemarket.comBlocks some AI searchblockedallowedallowedblockedallowedallowedblockedblocked–
8932mitsui-shopping-park.comOpenallowedallowedallowedallowedallowedallowedallowedallowed–
9075buyma.comOpenpartialallowedallowedpartialallowedallowedallowedallowed–
9516axis-marketplace.comOpenallowedallowedallowedallowedallowedallowedallowedallowed–
9546dismall.comUnreadableunknownunknownunknownunknownunknownunknownunknownunknown–
9656smallbiztrends.comOpenallowedallowedallowedallowedallowedallowedallowedallowed–
9896disneystore.comOpenallowedallowedallowedallowedallowedallowedallowedallowedyes
10000allbirds.comOpenallowedallowedallowedallowedallowedallowedallowedallowedyes

FAQ

Do e-commerce websites block AI crawlers?
As of 2026-11-16, 14.7% of the 116 e-commerce websites whose robots.txt we could read block GPTBot, 17.2% block ClaudeBot and 5.2% block PerplexityBot (all sites: 15.4%, 14.7%, 8.2%). 18 of 139 publish an llms.txt file.
Which e-commerce websites block GPTBot?
amazon.com, ebay.com, alibaba.com, aliexpress.com, bandcamp.com, rightmove.co.uk, cardmarket.com, videeyss.shop, bookshop.org, midasbuy.com, and more (full list on this page).

Other categories