AI Crawler Index
AI crawler rules of 10,000 popular websites
One page per site: which of 29 AI crawlers (GPTBot, ClaudeBot, PerplexityBot and more) its robots.txt blocks, whether it has an llms.txt, and what changed. Browse by category, by domain suffix or alphabetically. Checked daily, last on 2026-11-16.
By category
- Other 7,963 sites
- News & media 431 sites
- Entertainment 370 sites
- Government 330 sites
- Education 289 sites
- Tech 183 sites
- Finance 150 sites
- E-commerce 139 sites
- Social 56 sites
- Reference 53 sites
- Travel 14 sites
- Telecom 12 sites
- AI 10 sites
By country and domain suffix
- .com 5,298 sites
- .org 587 sites
- .net 409 sites
- .ru (Russia) 346 sites
- .de (Germany) 224 sites
- .jp (Japan) 220 sites
- .io 200 sites
- .edu 183 sites
- .gov 145 sites
- .br (Brazil) 128 sites
- .uk (United Kingdom) 116 sites
- .fr (France) 107 sites
- .in (India) 93 sites
- .cn (China) 92 sites
- .it (Italy) 81 sites
- .ai 64 sites
- .pl (Poland) 59 sites
- .au (Australia) 56 sites
- .tv 53 sites
- .nl (Netherlands) 53 sites
- .es (Spain) 52 sites
- .co 50 sites
- .ca (Canada) 49 sites
- .ch (Switzerland) 39 sites
- .ir (Iran) 39 sites
- .cz (Czechia) 37 sites
- .app 31 sites
A-Z
The 200 most popular sites
- google.com
- cloudflare.com
- facebook.com
- microsoft.com
- youtube.com
- apple.com
- instagram.com
- mail.ru
- twitter.com
- dzen.ru
- linkedin.com
- office.com
- amazon.com
- azure.com
- github.com
- wikipedia.org
- bing.com
- cloud.microsoft
- netflix.com
- gandi.net
- x.com
- wordpress.org
- skype.com
- icloud.com
- pinterest.com
- tiktok.com
- whatsapp.com
- yahoo.com
- roblox.com
- adobe.com
- spotify.com
- zoom.us
- qq.com
- nginx.org
- chatgpt.com
- opera.com
- vimeo.com
- openai.com
- f5.com
- sentry.io
- mozilla.org
- yandex.ru
- t.me
- samsung.com
- reddit.com
- ui.com
- europa.eu
- amazonvideo.com
- reg.ru
- snapchat.com
- apache.org
- dropbox.com
- intuit.com
- vk.com
- discord.gg
- unity3d.com
- kaspersky.com
- archive.org
- tumblr.com
- nytimes.com
- mts.ru
- paypal.com
- shopify.com
- vk.ru
- base.org
- w3.org
- flickr.com
- soundcloud.com
- creativecommons.org
- medium.com
- theguardian.com
- ebay.com
- android.com
- telekom.net
- t-online.de
- doi.org
- ubuntu.com
- forbes.com
- bbc.com
- twitch.tv
- mit.edu
- researchgate.net
- cisco.com
- telegram.org
- stripe.com
- nist.gov
- canva.com
- sourceforge.net
- cpanel.net
- wikimedia.org
- gmail.com
- hubspot.com
- roku.com
- who.int
- imdb.com
- namecheap.com
- launchpad.net
- booking.com
- mobile.de
- ibm.com
- reuters.com
- springer.com
- googleblog.com
- hp.com
- nature.com
- www.gov.uk
- steampowered.com
- weibo.com
- ok.ru
- debian.org
- harvard.edu
- alibaba.com
- weather.com
- wiley.com
- dell.com
- drom.ru
- crpt.ru
- php.net
- etsy.com
- aliyun.com
- name.com
- weebly.com
- bilibili.com
- gnu.org
- slack.com
- naver.com
- trustpilot.com
- ea.com
- checkpoint.com
- cdc.gov
- selectel.ru
- un.org
- washingtonpost.com
- tradingview.com
- gitlab.com
- nasa.gov
- zendesk.com
- nvidia.com
- issuu.com
- tp-link.com
- netangels.ru
- bloomberg.com
- stanford.edu
- wsj.com
- plesk.com
- calendly.com
- bsky.app
- espn.com
- unsplash.com
- globo.com
- 163.com
- line.me
- brave.com
- go.com
- claude.ai
- dailymotion.com
- steamcommunity.com
- aliexpress.com
- temu.com
- huawei.com
- supercell.com
- intel.com
- duckduckgo.com
- taobao.com
- paloaltonetworks.com
- substack.com
- duolingo.com
- businessinsider.com
- aol.com
- pixabay.com
- walmart.com
- arxiv.org
- speedtest.net
- goodreads.com
- amazonalexa.com
- mikrotik.com
- okta.com
- cnbc.com
- statista.com
- sina.com.cn
- ietf.org
- cloudflareinsights.com
- sohu.com
- quora.com
- addtoany.com
- ivi.ru
- shein.com
- bitdefender.net
- gosuslugi.ru
- mediatek.com
- docker.com
- scribd.com
- behance.net
- indiatimes.com
- foxnews.com
- gotinder.com
- noaa.gov
- eventbrite.com
- mysql.com
- share.google