AI Crawler Index · category
Do e-commerce websites block AI crawlers?
As of 2026-11-16, 14.7% of the 116 e-commerce websites whose robots.txt we could read block GPTBot, 17.2% block ClaudeBot and 5.2% block PerplexityBot (all sites: 15.4%, 14.7%, 8.2%). 18 of 139 publish an llms.txt file.
Block rate by crawler: e-commerce vs all sites
| Crawler | Operator | E-commerce | All sites |
|---|---|---|---|
| CCBot | Common Crawl | 18.1% | 16.4% |
| Bytespider | ByteDance | 18.1% | 16.2% |
| ClaudeBot | Anthropic | 17.2% | 14.7% |
| Amazonbot | Amazon | 17.2% | 13.2% |
| GPTBot | OpenAI | 14.7% | 15.4% |
| Meta-ExternalAgent | Meta | 14.7% | 13.6% |
| Google-Extended | 13.8% | 13.2% | |
| Applebot-Extended | Apple | 12.9% | 13.4% |
| cohere-ai | Cohere | 11.2% | 12.7% |
| MistralAI-Training | Mistral | 8.6% | 7.1% |
| Google-CloudVertexBot | 7.8% | 8.8% | |
| PerplexityBot | Perplexity | 5.2% | 8.2% |
| Claude-SearchBot | Anthropic | 4.3% | 5.7% |
| Claude-User | Anthropic | 4.3% | 5.9% |
| Perplexity-User | Perplexity | 4.3% | 6.1% |
| OAI-SearchBot | OpenAI | 3.4% | 5.8% |
| ChatGPT-User | OpenAI | 3.4% | 7.4% |
| OAI-AdsBot | OpenAI | 3.4% | 3% |
| meta-webindexer | Meta | 3.4% | 5.1% |
| meta-externalfetcher | Meta | 3.4% | 6.5% |
| Amzn-SearchBot | Amazon | 3.4% | 4.4% |
| DuckAssistBot | DuckDuckGo | 3.4% | 5.9% |
| MistralAI-User | Mistral | 3.4% | 5.7% |
| Google-GeminiNotebook | 2.6% | 3.3% | |
| Google-Agent | 2.6% | 3.6% | |
| meta-externalads | Meta | 2.6% | 2.9% |
| Amzn-User | Amazon | 2.6% | 4.4% |
| MistralAI-Index | Mistral | 2.6% | 3.6% |
| Applebot | Apple | 1.7% | 3.2% |
All 139 e-commerce sites in the index
✕ blocked · ◐ partially blocked (some paths) · ✓ allowed · ? robots.txt unreadable. "#" is the position in our popularity-ordered sample.
| # | Site | Policy | GPTBot | OAI-Search | ChatGPT-User | ClaudeBot | Claude-Search | Perplexity | Google-Ext. | CCBot | llms.txt |
|---|---|---|---|---|---|---|---|---|---|---|---|
| 13 | amazon.com | Blocks some AI search | blocked | blocked | blocked | blocked | blocked | blocked | blocked | blocked | – |
| 72 | shopify.com | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | yes |
| 86 | ebay.com | Blocks some AI search | blocked | partial | partial | blocked | partial | blocked | allowed | blocked | – |
| 101 | ozon.ru | Unreadable | unknown | unknown | unknown | unknown | unknown | unknown | unknown | unknown | – |
| 134 | alibaba.com | Blocks training only | blocked | allowed | allowed | allowed | allowed | allowed | blocked | allowed | – |
| 141 | etsy.com | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | yes |
| 188 | aliexpress.com | Blocks training only | blocked | allowed | allowed | blocked | allowed | allowed | blocked | blocked | – |
| 189 | temu.com | No robots.txt | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 194 | taobao.com | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 202 | walmart.com | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 218 | shein.com | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | yes |
| 235 | wildberries.ru | No robots.txt | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 252 | rakuten.co.jp | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 259 | shop.app | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | yes |
| 267 | target.com | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | yes |
| 269 | avito.ru | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 280 | ikea.com | Blocks training only | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 282 | nike.com | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 291 | shopee.com.br | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 306 | allegro.pl | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | partial | yes |
| 329 | bandcamp.com | Blocks training only | blocked | allowed | allowed | blocked | allowed | allowed | blocked | blocked | – |
| 339 | homedepot.com | Unreadable | unknown | unknown | unknown | unknown | unknown | unknown | unknown | unknown | – |
| 345 | zillow.com | Blocks training only | allowed | allowed | allowed | partial | partial | allowed | allowed | allowed | – |
| 348 | bestbuy.com | Open | partial | partial | partial | partial | allowed | partial | partial | partial | – |
| 351 | craigslist.org | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 354 | fiverr.com | Open | allowed | partial | partial | allowed | partial | partial | partial | allowed | yes |
| 381 | wb.ru | No robots.txt | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 395 | kleinanzeigen.de | Unreadable | unknown | unknown | unknown | unknown | unknown | unknown | unknown | unknown | – |
| 408 | coupang.com | Unreadable | unknown | unknown | unknown | unknown | unknown | unknown | unknown | unknown | – |
| 416 | mercadolivre.com.br | Unreadable | unknown | unknown | unknown | unknown | unknown | unknown | unknown | unknown | – |
| 423 | mercadolibre.com.ar | Unreadable | unknown | unknown | unknown | unknown | unknown | unknown | unknown | unknown | – |
| 425 | lowes.com | Unreadable | unknown | unknown | unknown | unknown | unknown | unknown | unknown | unknown | – |
| 428 | jd.com | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 431 | otto.de | Open | partial | partial | partial | allowed | allowed | partial | allowed | allowed | yes |
| 445 | bol.com | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 447 | costco.com | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 451 | flipkart.com | Blocks some AI search | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 469 | kohls.com | Unreadable | unknown | unknown | unknown | unknown | unknown | unknown | unknown | unknown | – |
| 470 | olx.com.br | Unreadable | unknown | unknown | unknown | unknown | unknown | unknown | unknown | unknown | – |
| 477 | themeforest.net | Blocks training only | allowed | allowed | allowed | allowed | allowed | allowed | allowed | blocked | – |
| 494 | redfin.com | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 498 | skroutz.gr | Blocks training only | partial | partial | partial | partial | partial | partial | partial | blocked | – |
| 511 | finn.no | Open | partial | partial | partial | partial | allowed | allowed | partial | partial | – |
| 520 | rustore.ru | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 552 | indiamart.com | Blocks training only | allowed | allowed | allowed | blocked | allowed | allowed | allowed | allowed | – |
| 556 | leboncoin.fr | Blocks training only | partial | partial | partial | partial | partial | partial | partial | partial | – |
| 558 | sahibinden.com | Unreadable | unknown | unknown | unknown | unknown | unknown | unknown | unknown | unknown | – |
| 567 | rightmove.co.uk | Blocks training only | blocked | allowed | allowed | allowed | allowed | allowed | allowed | blocked | – |
| 568 | tori.fi | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 569 | meesho.com | Unreadable | unknown | unknown | unknown | unknown | unknown | unknown | unknown | unknown | – |
| 571 | marktplaats.nl | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | yes |
| 575 | digikala.com | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 581 | idealista.com | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 582 | divar.ir | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 594 | argos.co.uk | Unreadable | unknown | unknown | unknown | unknown | unknown | unknown | unknown | unknown | – |
| 597 | emag.ro | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 603 | cdiscount.com | Unreadable | unknown | unknown | unknown | unknown | unknown | unknown | unknown | unknown | – |
| 624 | kaspi.kz | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 625 | justdial.com | Blocks training only | allowed | allowed | allowed | blocked | allowed | allowed | allowed | allowed | – |
| 631 | suumo.jp | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 640 | subito.it | Unreadable | unknown | unknown | unknown | unknown | unknown | unknown | unknown | unknown | – |
| 641 | mediaexpert.pl | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 656 | leroymerlin.fr | Unreadable | unknown | unknown | unknown | unknown | unknown | unknown | unknown | unknown | – |
| 676 | carsensor.net | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 678 | property24.com | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 689 | action.com | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 691 | otomoto.pl | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 704 | cardmarket.com | Blocks some AI search | blocked | allowed | allowed | blocked | allowed | allowed | blocked | blocked | – |
| 716 | markt.de | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | yes |
| 722 | torob.com | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 807 | googll.store | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 904 | marketwatch.com | Blocks some AI search | partial | partial | partial | blocked | blocked | blocked | blocked | blocked | – |
| 957 | disdarrummers.shop | Blocks some AI search | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 1038 | videeyss.shop | Blocks some AI search | blocked | allowed | allowed | blocked | allowed | allowed | blocked | blocked | – |
| 1320 | tmall.com | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 2045 | smallpdf.com | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 2154 | prestashop.com | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 2156 | capitaloneshopping.com | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | yes |
| 2231 | buymeacoffee.com | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 2238 | gmarket.co.kr | Blocks some AI search | allowed | blocked | blocked | blocked | blocked | blocked | blocked | blocked | – |
| 2877 | buyee.jp | Unreadable | unknown | unknown | unknown | unknown | unknown | unknown | unknown | unknown | – |
| 3106 | emarketer.com | Open | allowed | allowed | allowed | allowed | allowed | allowed | partial | allowed | – |
| 3141 | bookshop.org | Blocks training only | blocked | partial | partial | blocked | partial | partial | blocked | blocked | – |
| 3300 | momoshop.com.tw | No robots.txt | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 3331 | onthemarket.com | Blocks training only | allowed | allowed | allowed | blocked | allowed | allowed | allowed | allowed | – |
| 3445 | webstaurantstore.com | Unreadable | unknown | unknown | unknown | unknown | unknown | unknown | unknown | unknown | – |
| 3502 | trustedshops.com | Blocks training only | allowed | allowed | allowed | allowed | allowed | allowed | allowed | blocked | – |
| 3828 | comparethemarket.com | Unreadable | unknown | unknown | unknown | unknown | unknown | unknown | unknown | unknown | – |
| 4130 | midasbuy.com | Blocks all AI search | blocked | blocked | blocked | blocked | blocked | blocked | blocked | blocked | – |
| 4393 | marketscreener.com | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 4419 | shop-apotheke.com | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 4434 | codashop.com | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 4503 | bigbadtoystore.com | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 5041 | shopier.com | Unreadable | unknown | unknown | unknown | unknown | unknown | unknown | unknown | unknown | – |
| 5051 | litmarket.ru | Blocks training only | blocked | allowed | allowed | blocked | allowed | allowed | blocked | blocked | – |
| 5217 | shop-pro.jp | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | yes |
| 5428 | zalando.com | No robots.txt | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 5595 | naturabuy.fr | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 5633 | shopch.jp | No robots.txt | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 5651 | shopsy.in | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 5684 | shop-shimamura.com | Unreadable | unknown | unknown | unknown | unknown | unknown | unknown | unknown | unknown | – |
| 5711 | glorioushop.com | No robots.txt | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 5994 | sertastore.com.br | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 6139 | myshopline.com | No robots.txt | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 6179 | shopcider.com | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 6247 | market-qx.trade | Blocks some AI search | blocked | allowed | allowed | blocked | allowed | allowed | blocked | blocked | – |
| 6268 | jilateskrimpanas.store | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 6294 | makeshop.co.kr | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 6360 | videystream.store | Blocks some AI search | blocked | allowed | allowed | blocked | allowed | allowed | blocked | blocked | – |
| 6502 | rebuyengine.com | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 6504 | marketsandmarkets.com | Unreadable | unknown | unknown | unknown | unknown | unknown | unknown | unknown | unknown | – |
| 6713 | researchandmarkets.com | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 6823 | wholefoodsmarket.com | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 6855 | yamallng.ru | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 6862 | chiikawamarket.jp | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | yes |
| 7139 | mencretmesum.store | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 7165 | letyshops.com | Blocks some AI search | blocked | blocked | blocked | blocked | blocked | blocked | blocked | blocked | – |
| 7259 | marketiq.com | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 7469 | 360buyimg.com | No robots.txt | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 7600 | tiktokshop.com | No robots.txt | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 7735 | viideyss.store | Blocks some AI search | blocked | allowed | allowed | blocked | allowed | allowed | blocked | blocked | – |
| 7811 | dmarket.com | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | yes |
| 8168 | storeland.ru | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 8234 | moneysupermarket.com | Open | partial | partial | partial | partial | partial | partial | allowed | partial | yes |
| 8236 | myshoptet.com | Unreadable | unknown | unknown | unknown | unknown | unknown | unknown | unknown | unknown | – |
| 8259 | shopmy.us | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | yes |
| 8454 | contentmarketinginstitute.com | Blocks training only | blocked | allowed | allowed | blocked | allowed | allowed | allowed | blocked | – |
| 8461 | fakestoreapi.com | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 8567 | shopbop.com | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 8803 | marketplace.org | No robots.txt | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 8883 | stores.jp | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 8901 | creativemarket.com | Blocks some AI search | blocked | allowed | allowed | blocked | allowed | allowed | blocked | blocked | – |
| 8932 | mitsui-shopping-park.com | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 9075 | buyma.com | Open | partial | allowed | allowed | partial | allowed | allowed | allowed | allowed | – |
| 9516 | axis-marketplace.com | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 9546 | dismall.com | Unreadable | unknown | unknown | unknown | unknown | unknown | unknown | unknown | unknown | – |
| 9656 | smallbiztrends.com | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 9896 | disneystore.com | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | yes |
| 10000 | allbirds.com | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | yes |
FAQ
- Do e-commerce websites block AI crawlers?
- As of 2026-11-16, 14.7% of the 116 e-commerce websites whose robots.txt we could read block GPTBot, 17.2% block ClaudeBot and 5.2% block PerplexityBot (all sites: 15.4%, 14.7%, 8.2%). 18 of 139 publish an llms.txt file.
- Which e-commerce websites block GPTBot?
- amazon.com, ebay.com, alibaba.com, aliexpress.com, bandcamp.com, rightmove.co.uk, cardmarket.com, videeyss.shop, bookshop.org, midasbuy.com, and more (full list on this page).
Other categories
- AI GPTBot 10%
- Education GPTBot 5.6%
- Entertainment GPTBot 19.6%
- Finance GPTBot 6.9%
- Government GPTBot 3%
- News & media GPTBot 33.7%
- Other GPTBot 15.1%
- Reference GPTBot 27.9%
- Social GPTBot 37.3%
- Tech GPTBot 3.7%
- Telecom GPTBot 0%
- Travel GPTBot 15.4%