AI Crawler Index
AI crawler rules of 1,005 popular websites
One page per site: which of 29 AI crawlers (GPTBot, ClaudeBot, PerplexityBot, Google-Extended, CCBot and more) its robots.txt allows or blocks, llms.txt and Content-Signal, checked daily. Last check: 2026-10-01.
AI 9
E-commerce 73
- amazon.com
- shopify.com
- ebay.com
- ozon.ru
- alibaba.com
- etsy.com
- aliexpress.com
- temu.com
- taobao.com
- walmart.com
- shein.com
- wildberries.ru
- rakuten.co.jp
- shop.app
- target.com
- avito.ru
- ikea.com
- nike.com
- shopee.com.br
- allegro.pl
- bandcamp.com
- homedepot.com
- zillow.com
- bestbuy.com
- craigslist.org
- fiverr.com
- wb.ru
- kleinanzeigen.de
- coupang.com
- mercadolivre.com.br
- mercadolibre.com.ar
- lowes.com
- jd.com
- otto.de
- flipkart.com
- costco.com
- kohls.com
- olx.com.br
- themeforest.net
- skroutz.gr
- redfin.com
- finn.no
- indiamart.com
- leboncoin.fr
- sahibinden.com
- rightmove.co.uk
- meesho.com
- marktplaats.nl
- tori.fi
- digikala.com
- divar.ir
- idealista.com
- argos.co.uk
- emag.ro
- cdiscount.com
- justdial.com
- kaspi.kz
- suumo.jp
- subito.it
- mediaexpert.pl
- leroymerlin.fr
- carsensor.net
- property24.com
- action.com
- otomoto.pl
- cardmarket.com
- markt.de
- torob.com
- googll.store
- marketwatch.com
- disdarrummers.shop
- zalando.com
- allbirds.com
Education 24
Entertainment 77
- youtube.com
- netflix.com
- roblox.com
- spotify.com
- vimeo.com
- amazonvideo.com
- epicgames.com
- soundcloud.com
- twitch.tv
- steampowered.com
- bilibili.com
- ea.com
- dailymotion.com
- steamcommunity.com
- supercell.com
- ivi.ru
- playstation.com
- nintendo.com
- primevideo.com
- kick.com
- itch.io
- imgur.com
- curseforge.com
- warnerbros.com
- ted.com
- disneyplus.com
- xbox.com
- chess.com
- youku.com
- playrix.com
- nexusmods.com
- ubi.com
- pixiv.net
- flightradar24.com
- poki.com
- crazygames.com
- riotgames.com
- tabelog.com
- aparat.com
- giphy.com
- cookpad.com
- tbs.co.jp
- nexon.com
- lichess.org
- pandora.com
- rutube.ru
- battle.net
- netkeiba.com
- playhop.com
- lookmovie2.to
- aternos.org
- cardgames.io
- mangabuff.ru
- filmweb.pl
- zedge.net
- gamewith.jp
- myinstants.com
- hanime1.me
- musicdel.ir
- playfabapi.com
- myreadingmanga.info
- moe.video
- plex.tv
- kingbokep.tv
- nustargame.com
- skipthegames.com
- arionplay.com
- yasyadong02.tv
- mangaraw.ac
- playinboss.com
- kolkataff.tv
- filmyfly.soy
- pluto.tv
- hulu.com
- iqiyi.com
- gamebanana.com
- videoxx.com
Finance 21
Government 52
- europa.eu
- nih.gov
- miit.gov.cn
- nist.gov
- www.gov.uk
- cdc.gov
- un.org
- nasa.gov
- beian.gov.cn
- noaa.gov
- ca.gov
- unesco.org
- loc.gov
- service.gov.uk
- fda.gov
- usda.gov
- worldbank.org
- epa.gov
- state.gov
- caixa.gov.br
- weforum.org
- whitehouse.gov
- maricopa.gov
- oecd.org
- acesso.gov.br
- canada.ca
- admin.ch
- rajasthan.gov.in
- gst.gov.in
- irs.gov
- turkiye.gov.tr
- incometax.gov.in
- inps.it
- epfindia.gov.in
- nfse.gov.br
- eci.gov.in
- parivahan.gov.in
- afip.gob.ar
- osym.gov.tr
- ftc.gov
- jra.go.jp
- kemkes.go.id
- digitallocker.gov.in
- dord.gov.in
- cbl.gov.ly
- va.gov
- nps.gov
- usgs.gov
- sec.gov
- meb.gov.tr
- ed.gov
- bls.gov
News & media 139
- nytimes.com
- medium.com
- cnn.com
- theguardian.com
- t-online.de
- forbes.com
- bbc.com
- reuters.com
- weather.com
- tradingview.com
- bloomberg.com
- wsj.com
- espn.com
- globo.com
- substack.com
- businessinsider.com
- cnbc.com
- statista.com
- indiatimes.com
- foxnews.com
- ft.com
- wired.com
- time.com
- uol.com.br
- usatoday.com
- telegraph.co.uk
- accuweather.com
- techcrunch.com
- elpais.com
- dailymail.com
- cbsnews.com
- lemonde.fr
- cnet.com
- independent.co.uk
- wp.pl
- bild.de
- mlb.com
- nikkei.com
- apnews.com
- welt.de
- prnewswire.com
- people.com
- lefigaro.fr
- theverge.com
- focus.de
- hbr.org
- nbcnews.com
- latimes.com
- onet.pl
- repubblica.it
- corriere.it
- investopedia.com
- elmundo.es
- nationalgeographic.com
- flashscore.com
- infobae.com
- theatlantic.com
- moneycontrol.com
- pbs.org
- n-tv.de
- ouest-france.fr
- interia.pl
- goal.com
- terra.com.br
- as.com
- bfmtv.com
- huffpost.com
- marca.com
- skysports.com
- wetteronline.de
- aftonbladet.se
- gsmarena.com
- lanacion.com.ar
- hltv.org
- nypost.com
- clarin.com
- economist.com
- theconversation.com
- ad.nl
- wetter.com
- itmedia.co.jp
- nu.nl
- elconfidencial.com
- vg.no
- spiegel.de
- cricinfo.com
- lequipe.fr
- rbc.ru
- gazzetta.it
- hln.be
- mundodeportivo.com
- abc.net.au
- livescore.com
- fortune.com
- ladepeche.fr
- cricbuzz.com
- actu.fr
- ilfattoquotidiano.it
- meteofrance.com
- tenki.jp
- weathernews.jp
- varzesh3.com
- sport.es
- oricon.co.jp
- haberler.com
- nikkansports.com
- businesswire.com
- huffingtonpost.com
- novinky.cz
- dagbladet.no
- okdiario.com
- ilmeteo.it
- sapo.pt
- cronista.com
- dw.com
- iltalehti.fi
- protothema.gr
- aljazeera.com
- blic.rs
- jutarnji.hr
- 24h.com.vn
- index.hr
- sozcu.com.tr
- 3bmeteo.com
- jagran.com
- ekstrabladet.dk
- eenadu.net
- kooora.com
- sondakika.com
- meteociel.fr
- sportfilm800.com
- newsweek.com
- arstechnica.com
- sciencedaily.com
- zdnet.com
- euronews.com
- washingtonpost.com
- engadget.com
- dev.to
Other 305
- azure.com
- cloud.microsoft
- gandi.net
- windows.com
- dellemc.com
- reg.ru
- base.org
- hosting24.com
- macromedia.com
- flickr.com
- creativecommons.org
- roku.com
- launchpad.net
- mobile.de
- googleblog.com
- drom.ru
- crpt.ru
- aliyun.com
- name.com
- trustpilot.com
- selectel.ru
- netangels.ru
- indeed.com
- timeweb.ru
- go.com
- amazonalexa.com
- cloudflareinsights.com
- bitdefender.net
- gosuslugi.ru
- eventbrite.com
- share.google
- capcut.com
- bitrix24.ru
- blogger.com
- oraclecloud.com
- ovh.net
- yelp.com
- vkontakte.ru
- uk.com
- wpguardian.io
- outbrain.com
- consultant.ru
- fb.com
- porkbun.com
- oxylabs.io
- webempresa.eu
- beget.ru
- discordapp.com
- usps.com
- pccc.com
- life360.com
- pendo.io
- it.com
- agora.io
- deloitte.com
- smartthings.com
- eu.com
- kontur.ru
- safety.google
- lge.com
- superhosting.bg
- inven.co.kr
- one.com
- inleed.net
- locaweb.com.br
- tabor.ru
- us.com
- jusbrasil.com.br
- ap4r.com
- beboo.ru
- litnet.com
- bbb.org
- naukri.com
- id.me
- grow.google
- edna.ru
- labs.google
- bookmyshow.com
- mbga.jp
- hostpoint.ch
- jino.ru
- sotwe.com
- cmoa.jp
- nosv.org
- platform.sh
- testbook.com
- 1mg.com
- xhwide.com
- immobiliare.it
- programme-tv.net
- bt.dk
- absher.sa
- notion.com
- aajtak.in
- manulife-sinochem.com
- iau.ir
- ppomppu.co.kr
- goodreturns.in
- kakuyomu.jp
- deezer.com
- rakuten-sec.co.jp
- tjk.org
- buzzfeed.com
- bikewale.com
- mongodb.com
- trilltrill.jp
- aka.ms
- therapservices.net
- autoplius.lt
- theqoo.net
- smadex.com
- screener.in
- rayjump.com
- mamastar.jp
- acm.org
- tupi.fm
- chordtela.com
- wpastra.com
- girlschannel.net
- freewebnovel.com
- czbooks.net
- mechacomic.jp
- ss.com
- skelbiu.lt
- rambler.ru
- piccoma.com
- eepurl.com
- alphapolis.co.jp
- lecturas.com
- letras.mus.br
- publi24.ro
- site.com
- hik-connect.com
- afribaba.com
- animanch.com
- dto.jp
- ecnavi.jp
- twkan.com
- arbada.com
- goal7.co
- eff.org
- promiedos.com.ar
- tjsp.jus.br
- turbo.az
- bancodevenezuela.com
- humoruniv.com
- choiceadvantage.com
- sun.com
- site777.jp
- az24.vn
- linkmake.in
- pagibigfundservices.com
- 0xrpc.io
- staseraintv.com
- sambad.com
- mobcup.fm
- rsch.jp
- shutterstock.com
- ad-contents.jp
- ico.org.uk
- rocketmoney.dev
- gartner.com
- akismet.com
- enable-javascript.com
- hover.com
- overwolf.com
- waze.com
- avira.com
- qcloud.com
- ubnt.com
- fao.org
- tawk.to
- kargo.com
- amocrm.ru
- contextweb.com
- vmware.com
- w3schools.com
- beeline.ru
- medallia.com
- blizzard.com
- acs.org
- adroll.com
- rapid7.com
- wordwall.net
- newyorker.com
- fc2.com
- fedex.com
- trello.com
- register.com
- kraken.com
- 1c.ru
- moodle.org
- blackhub.team
- pewresearch.org
- trustarc.com
- loopia.se
- vinted.fr
- verify.hn
- fifa.com
- gdemoideti.ru
- sweb.ru
- pusher.com
- yimg.com
- sitescout.com
- adrta.com
- sky.com
- elegantthemes.com
- scientificamerican.com
- indexww.com
- clevelandclinic.org
- cpmstar.com
- xinhuanet.com
- norton.com
- orcid.org
- wayground.com
- ixl.com
- vice.com
- merkur.de
- samsungknox.com
- 1password.com
- fb.watch
- astrumu.io
- sciencemag.org
- thelancet.com
- charter.net
- www.nhs.uk
- mapbox.com
- sprinthost.ru
- wikihow.com
- git-scm.com
- oreilly.com
- appier.net
- deepintent.com
- iubenda.com
- bigcommerce.com
- themeisle.com
- moengage.com
- hh.ru
- hetzner.com
- autotrader.co.uk
- squareup.com
- mindbox.ru
- hola.com
- ueiwsp.com
- salthasher.com
- wa.link
- jamanetwork.com
- fastcompany.com
- strava.com
- pnas.org
- ninjarmm.com
- livechatinc.com
- threema.ch
- mitre.org
- auone.jp
- m.me
- dantri.com.vn
- richaudience.com
- rackspace.com
- techtarget.com
- liquipedia.net
- lnkd.in
- iscorp.com
- ssrn.com
- wpengine.com
- springserve.com
- romarg.com
- privacymanager.io
- freebsd.org
- sogou.com
- logitech.com
- connatix.com
- vlr.gg
- umeng.com
- russianfood.com
- intentiq.com
- truste.com
- acrobat.com
- typepad.com
- science.org
- dondominio.com
- granbluefantasy.jp
- wechat.com
- business.site
- getbootstrap.com
- about.com
- sakura.ne.jp
- ign.com
- wunderground.com
- iponweb.net
- acronis.com
- suruga-ya.jp
- matterport.com
- scryfall.com
- eltiempo.es
- earthlink.net
- gizmodo.com
- salesforceliveagent.com
- automattic.com
- npmjs.com
- clickpetroleoegas.com.br
- timeanddate.com
- broadcom.com
- alura.com.br
- sedoparking.com
Reference 53
- wikipedia.org
- archive.org
- w3.org
- doi.org
- sciencedirect.com
- researchgate.net
- wikimedia.org
- who.int
- imdb.com
- springer.com
- nature.com
- wiley.com
- goodreads.com
- arxiv.org
- ietf.org
- quora.com
- stackoverflow.com
- fandom.com
- oup.com
- tandfonline.com
- openstreetmap.org
- ieee.org
- mdpi.com
- sagepub.com
- britannica.com
- cambridge.org
- iso.org
- merriam-webster.com
- academia.edu
- frontiersin.org
- biblegateway.com
- mayoclinic.org
- genius.com
- webmd.com
- healthline.com
- rfc-editor.org
- letterboxd.com
- apa.org
- ultimate-guitar.com
- dafont.com
- plos.org
- jw.org
- namu.wiki
- wiktionary.org
- themoviedb.org
- psychologytoday.com
- mydramalist.com
- bmj.com
- jstor.org
- calculator.net
- biomedcentral.com
- elsevier.com
- stackexchange.com
Social 54
- facebook.com
- instagram.com
- twitter.com
- linkedin.com
- x.com
- skype.com
- pinterest.com
- tiktok.com
- whatsapp.com
- t.me
- reddit.com
- snapchat.com
- vk.com
- discord.gg
- tumblr.com
- vk.ru
- telegram.org
- weibo.com
- bsky.app
- line.me
- behance.net
- gotinder.com
- deviantart.com
- threads.com
- viber.com
- patreon.com
- meta.com
- wattpad.com
- ameblo.jp
- livejournal.com
- pikabu.ru
- myspace.com
- note.com
- change.org
- xing.com
- ameba.jp
- archiveofourown.org
- gofundme.com
- kickstarter.com
- fmkorea.com
- dcinside.com
- livedoor.com
- dribbble.com
- fanfiction.net
- douyin.com
- arca.live
- ruliweb.com
- syosetu.com
- ficbook.net
- badoo.com
- ptt.cc
- ninisite.com
- dogdrip.net
- meetup.com
Tech 173
- google.com
- cloudflare.com
- microsoft.com
- apple.com
- mail.ru
- office.com
- github.com
- bing.com
- wordpress.org
- icloud.com
- yahoo.com
- adobe.com
- qq.com
- zoom.us
- nginx.org
- baidu.com
- opera.com
- f5.com
- sentry.io
- samsung.com
- mozilla.org
- ui.com
- office365.com
- apache.org
- dropbox.com
- unity3d.com
- kaspersky.com
- godaddy.com
- xiaomi.com
- webex.com
- oracle.com
- android.com
- ubuntu.com
- cisco.com
- stripe.com
- canva.com
- cpanel.net
- sourceforge.net
- meraki.com
- gmail.com
- hubspot.com
- namecheap.com
- ibm.com
- hp.com
- autodesk.com
- debian.org
- avast.com
- dell.com
- php.net
- hostinger.com
- mi.com
- gnu.org
- weebly.com
- arubanetworks.com
- naver.com
- slack.com
- checkpoint.com
- gitlab.com
- hostgator.com
- zendesk.com
- nvidia.com
- tp-link.com
- issuu.com
- plesk.com
- calendly.com
- unsplash.com
- 163.com
- brave.com
- netease.com
- huawei.com
- paloaltonetworks.com
- duckduckgo.com
- duolingo.com
- aol.com
- pixabay.com
- bluehost.com
- speedtest.net
- mikrotik.com
- sina.com.cn
- sohu.com
- okta.com
- addtoany.com
- mediatek.com
- docker.com
- scribd.com
- mysql.com
- slideshare.net
- trendmicro.com
- squarespace.com
- firefox.com
- fast.com
- digitalocean.com
- grammarly.com
- eset.com
- free.fr
- teamviewer.com
- dreamhost.com
- klaviyo.com
- atlassian.com
- wix.com
- tencent.com
- mega.co.nz
- jotform.com
- linode.com
- intercom.io
- redhat.com
- 360.cn
- surveymonkey.com
- anydesk.com
- sophos.com
- wps.com
- ring.com
- seznam.cz
- ya.ru
- asus.com
- mailchimp.com
- python.org
- mediafire.com
- box.com
- lenovo.com
- vivo.com.cn
- zoho.com
- odoo.com
- me.com
- blackberry.com
- daum.net
- pexels.com
- jetbrains.com
- figma.com
- synology.com
- cloudinary.com
- disqus.com
- lg.com
- arcgis.com
- starlink.com
- qualtrics.com
- tailscale.com
- xerox.com
- statcounter.com
- jimdo.com
- ecosia.org
- nextcloud.com
- spamhaus.org
- typeform.com
- freepik.com
- garmin.com
- supabase.co
- adblockplus.org
- netgear.com
- ilovepdf.com
- instructure.com
- magnific.com
- service-now.com
- meethue.com
- auth0.com
- tesla.com
- classlink.com
- siemens.com
- remove.bg
- coursera.org
- savefrom.net
- kakao.com
- ovhcloud.com
- postgresql.org
- clever.com
- fdown.net
- kernel.org
- twilio.com
- rustdesk.com
- ssstik.io
- utorrent.com
- fedoraproject.org
- udemy.com