AI Crawler Index · category
Do tech websites block AI crawlers?
As of 2026-11-16, 3.7% of the 162 tech websites whose robots.txt we could read block GPTBot, 5.6% block ClaudeBot and 1.2% block PerplexityBot (all sites: 15.4%, 14.7%, 8.2%). 62 of 183 publish an llms.txt file.
Block rate by crawler: tech vs all sites
| Crawler | Operator | Tech | All sites |
|---|---|---|---|
| Bytespider | ByteDance | 8% | 16.2% |
| CCBot | Common Crawl | 6.2% | 16.4% |
| ClaudeBot | Anthropic | 5.6% | 14.7% |
| GPTBot | OpenAI | 3.7% | 15.4% |
| Google-Extended | 2.5% | 13.2% | |
| Applebot-Extended | Apple | 1.9% | 13.4% |
| cohere-ai | Cohere | 1.9% | 12.7% |
| PerplexityBot | Perplexity | 1.2% | 8.2% |
| Meta-ExternalAgent | Meta | 1.2% | 13.6% |
| ChatGPT-User | OpenAI | 0.6% | 7.4% |
| OAI-AdsBot | OpenAI | 0.6% | 3% |
| Claude-SearchBot | Anthropic | 0.6% | 5.7% |
| Google-GeminiNotebook | 0.6% | 3.3% | |
| meta-webindexer | Meta | 0.6% | 5.1% |
| meta-externalads | Meta | 0.6% | 2.9% |
| Amazonbot | Amazon | 0.6% | 13.2% |
| Amzn-SearchBot | Amazon | 0.6% | 4.4% |
| Amzn-User | Amazon | 0.6% | 4.4% |
| MistralAI-Index | Mistral | 0.6% | 3.6% |
| MistralAI-Training | Mistral | 0.6% | 7.1% |
| OAI-SearchBot | OpenAI | 0% | 5.8% |
| Claude-User | Anthropic | 0% | 5.9% |
| Perplexity-User | Perplexity | 0% | 6.1% |
| Google-CloudVertexBot | 0% | 8.8% | |
| Google-Agent | 0% | 3.6% | |
| Applebot | Apple | 0% | 3.2% |
| meta-externalfetcher | Meta | 0% | 6.5% |
| DuckAssistBot | DuckDuckGo | 0% | 5.9% |
| MistralAI-User | Mistral | 0% | 5.7% |
Top 150 of 183 tech sites in the index
✕ blocked · ◐ partially blocked (some paths) · ✓ allowed · ? robots.txt unreadable. "#" is the position in our popularity-ordered sample.
| # | Site | Policy | GPTBot | OAI-Search | ChatGPT-User | ClaudeBot | Claude-Search | Perplexity | Google-Ext. | CCBot | llms.txt |
|---|---|---|---|---|---|---|---|---|---|---|---|
| 1 | google.com | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 2 | cloudflare.com | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | yes |
| 4 | microsoft.com | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 6 | apple.com | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 8 | mail.ru | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 12 | office.com | No robots.txt | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 15 | github.com | Blocks training only | partial | partial | partial | partial | partial | partial | allowed | allowed | yes |
| 17 | bing.com | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 22 | wordpress.org | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | yes |
| 24 | icloud.com | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 28 | yahoo.com | Blocks some AI search | blocked | partial | blocked | blocked | partial | blocked | blocked | blocked | – |
| 30 | adobe.com | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | yes |
| 32 | zoom.us | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 33 | qq.com | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 34 | nginx.org | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 35 | baidu.com | Unreadable | unknown | unknown | unknown | unknown | unknown | unknown | unknown | unknown | – |
| 37 | opera.com | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | yes |
| 40 | f5.com | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 41 | sentry.io | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | yes |
| 42 | mozilla.org | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 43 | yandex.ru | Open | allowed | partial | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 46 | samsung.com | Open | partial | partial | partial | allowed | allowed | partial | partial | allowed | yes |
| 49 | ui.com | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 50 | office365.com | Unreadable | unknown | unknown | unknown | unknown | unknown | unknown | unknown | unknown | – |
| 56 | apache.org | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 57 | dropbox.com | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | yes |
| 62 | unity3d.com | Blocks training only | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 63 | kaspersky.com | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | yes |
| 65 | godaddy.com | Unreadable | unknown | unknown | unknown | unknown | unknown | unknown | unknown | unknown | – |
| 68 | xiaomi.com | Unreadable | unknown | unknown | unknown | unknown | unknown | unknown | unknown | unknown | – |
| 70 | webex.com | Unreadable | unknown | unknown | unknown | unknown | unknown | unknown | unknown | unknown | yes |
| 87 | android.com | Open | partial | partial | allowed | partial | partial | partial | partial | partial | – |
| 91 | oracle.com | Unreadable | unknown | unknown | unknown | unknown | unknown | unknown | unknown | unknown | – |
| 93 | ubuntu.com | Open | partial | partial | partial | partial | partial | partial | partial | allowed | yes |
| 100 | cisco.com | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | yes |
| 104 | stripe.com | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | yes |
| 106 | canva.com | Blocks training only | blocked | allowed | allowed | blocked | allowed | allowed | allowed | blocked | – |
| 107 | sourceforge.net | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 108 | cpanel.net | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 110 | meraki.com | Unreadable | unknown | unknown | unknown | unknown | unknown | unknown | unknown | unknown | – |
| 111 | gmail.com | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 112 | hubspot.com | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | yes |
| 116 | namecheap.com | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 120 | ibm.com | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 124 | hp.com | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 125 | autodesk.com | Unreadable | unknown | unknown | unknown | unknown | unknown | unknown | unknown | unknown | – |
| 130 | avast.com | Unreadable | unknown | unknown | unknown | unknown | unknown | unknown | unknown | unknown | – |
| 132 | debian.org | No robots.txt | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 137 | dell.com | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | yes |
| 140 | php.net | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 144 | mi.com | Unreadable | unknown | unknown | unknown | unknown | unknown | unknown | unknown | unknown | – |
| 145 | weebly.com | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 147 | gnu.org | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 148 | arubanetworks.com | Unreadable | unknown | unknown | unknown | unknown | unknown | unknown | unknown | unknown | – |
| 149 | slack.com | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | yes |
| 150 | naver.com | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 153 | checkpoint.com | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | yes |
| 157 | hostgator.com | Unreadable | unknown | unknown | unknown | unknown | unknown | unknown | unknown | unknown | – |
| 161 | gitlab.com | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 163 | zendesk.com | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | yes |
| 164 | nvidia.com | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | yes |
| 165 | issuu.com | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 166 | tp-link.com | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 171 | plesk.com | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | yes |
| 172 | calendly.com | Blocks training only | allowed | allowed | allowed | allowed | allowed | allowed | allowed | blocked | yes |
| 176 | unsplash.com | Blocks some AI search | allowed | allowed | allowed | blocked | blocked | allowed | allowed | allowed | – |
| 178 | 163.com | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 181 | brave.com | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 184 | netease.com | Unreadable | unknown | unknown | unknown | unknown | unknown | unknown | unknown | unknown | – |
| 190 | huawei.com | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 192 | intel.com | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | yes |
| 193 | duckduckgo.com | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 195 | paloaltonetworks.com | Open | allowed | partial | partial | allowed | partial | partial | allowed | allowed | yes |
| 197 | bluehost.com | Unreadable | unknown | unknown | unknown | unknown | unknown | unknown | unknown | unknown | yes |
| 198 | duolingo.com | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 200 | aol.com | Blocks training only | blocked | allowed | partial | blocked | allowed | partial | blocked | blocked | – |
| 201 | pixabay.com | Blocks training only | blocked | allowed | allowed | blocked | allowed | allowed | blocked | blocked | – |
| 204 | speedtest.net | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 207 | mikrotik.com | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 208 | okta.com | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | yes |
| 211 | sina.com.cn | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 214 | sohu.com | Blocks some AI search | partial | partial | partial | partial | partial | partial | partial | partial | – |
| 216 | addtoany.com | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 221 | mediatek.com | No robots.txt | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 223 | docker.com | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 224 | scribd.com | Blocks training only | allowed | allowed | allowed | blocked | allowed | allowed | allowed | blocked | – |
| 232 | mysql.com | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 234 | slideshare.net | Blocks training only | allowed | allowed | allowed | blocked | allowed | allowed | allowed | blocked | – |
| 239 | trendmicro.com | Open | allowed | partial | partial | partial | partial | partial | partial | partial | yes |
| 240 | squarespace.com | Open | partial | allowed | allowed | partial | allowed | allowed | partial | partial | yes |
| 243 | firefox.com | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 245 | fast.com | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 246 | digitalocean.com | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | yes |
| 247 | grammarly.com | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 254 | eset.com | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 255 | free.fr | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 258 | dreamhost.com | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | yes |
| 260 | teamviewer.com | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 266 | klaviyo.com | Open | partial | partial | partial | partial | partial | partial | allowed | allowed | yes |
| 272 | atlassian.com | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | yes |
| 278 | wix.com | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | yes |
| 281 | tencent.com | Unreadable | unknown | unknown | unknown | unknown | unknown | unknown | unknown | unknown | – |
| 283 | linode.com | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 285 | jotform.com | Open | partial | allowed | allowed | partial | allowed | partial | allowed | partial | – |
| 286 | mega.co.nz | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 290 | intercom.io | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | yes |
| 293 | redhat.com | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | yes |
| 297 | 360.cn | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 299 | surveymonkey.com | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | yes |
| 302 | anydesk.com | Unreadable | unknown | unknown | unknown | unknown | unknown | unknown | unknown | unknown | – |
| 303 | sophos.com | Blocks training only | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | yes |
| 318 | wps.com | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | yes |
| 319 | ring.com | Open | partial | allowed | partial | partial | allowed | partial | partial | allowed | yes |
| 322 | seznam.cz | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 324 | ya.ru | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 330 | asus.com | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | yes |
| 333 | mailchimp.com | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | yes |
| 335 | python.org | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 341 | mediafire.com | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 344 | box.com | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 350 | lenovo.com | Unreadable | unknown | unknown | unknown | unknown | unknown | unknown | unknown | unknown | – |
| 364 | zoho.com | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | yes |
| 366 | vivo.com.cn | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 370 | odoo.com | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 372 | me.com | No robots.txt | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 375 | blackberry.com | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 382 | pexels.com | Blocks training only | blocked | allowed | allowed | blocked | allowed | allowed | allowed | blocked | – |
| 384 | daum.net | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 386 | jetbrains.com | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 388 | figma.com | Blocks training only | blocked | partial | partial | blocked | partial | partial | blocked | blocked | – |
| 391 | cloudinary.com | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | yes |
| 392 | synology.com | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 400 | disqus.com | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 403 | lg.com | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | yes |
| 409 | arcgis.com | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 412 | starlink.com | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 414 | qualtrics.com | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 424 | tailscale.com | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | yes |
| 437 | xerox.com | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 452 | statcounter.com | No robots.txt | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 455 | jimdo.com | Open | partial | partial | partial | partial | partial | partial | partial | partial | yes |
| 460 | spamhaus.org | Unreadable | unknown | unknown | unknown | unknown | unknown | unknown | unknown | unknown | – |
| 461 | ecosia.org | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 462 | nextcloud.com | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 473 | typeform.com | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | yes |
| 474 | garmin.com | Blocks some AI search | allowed | allowed | allowed | allowed | allowed | blocked | allowed | allowed | – |
| 476 | freepik.com | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 481 | supabase.co | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | yes |
| 482 | instructure.com | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | yes |
| 483 | adblockplus.org | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
Sorted by popularity. Every site has its own page: all 10,000 sites (A-Z lists), or download the CSV.
FAQ
- Do tech websites block AI crawlers?
- As of 2026-11-16, 3.7% of the 162 tech websites whose robots.txt we could read block GPTBot, 5.6% block ClaudeBot and 1.2% block PerplexityBot (all sites: 15.4%, 14.7%, 8.2%). 62 of 183 publish an llms.txt file.
- Which tech websites block GPTBot?
- yahoo.com, canva.com, aol.com, pixabay.com, pexels.com, figma.com.
Other categories
- AI GPTBot 10%
- E-commerce GPTBot 14.7%
- Education GPTBot 5.6%
- Entertainment GPTBot 19.6%
- Finance GPTBot 6.9%
- Government GPTBot 3%
- News & media GPTBot 33.7%
- Other GPTBot 15.1%
- Reference GPTBot 27.9%
- Social GPTBot 37.3%
- Telecom GPTBot 0%
- Travel GPTBot 15.4%