AI Crawler Index · llms.txt
Which popular websites have an llms.txt?
As of 2026-11-16, 1,391 of 8,313 popular websites (16.7%) serve an llms.txt file, the plain-text guide for AI models; among the top 100 sites it is 16 of 83. Sites with an llms.txt block GPTBot 5.9% of the time, sites without one 17.3%.
1,391
sites with llms.txt
16.7%
of 8,313 readable sites
16
of the top 83
llms.txt adoption by category
| Category | Sites read | With llms.txt | Share |
|---|---|---|---|
| Tech | 162 | 60 | 37% |
| Finance | 102 | 23 | 22.5% |
| Other | 6,651 | 1,180 | 17.7% |
| E-commerce | 116 | 18 | 15.5% |
| News & media | 392 | 42 | 10.7% |
| Entertainment | 332 | 30 | 9% |
| Education | 232 | 14 | 6% |
| Social | 51 | 2 | 3.9% |
| Government | 203 | 8 | 3.9% |
| Reference | 43 | 1 | 2.3% |
llms.txt adoption by country and suffix
| Suffix | Sites read | With llms.txt | Share |
|---|---|---|---|
| .ai | 61 | 30 | 49.2% |
| .io | 186 | 67 | 36% |
| .app | 30 | 8 | 26.7% |
| .com | 4,449 | 941 | 21.2% |
| .br (Brazil) | 86 | 17 | 19.8% |
| .au (Australia) | 35 | 5 | 14.3% |
| .in (India) | 71 | 10 | 14.1% |
| .ru (Russia) | 288 | 33 | 11.5% |
| .it (Italy) | 52 | 6 | 11.5% |
| .co | 44 | 5 | 11.4% |
| .ir (Iran) | 35 | 4 | 11.4% |
| .uk (United Kingdom) | 91 | 10 | 11% |
| .org | 526 | 52 | 9.9% |
| .ca (Canada) | 42 | 4 | 9.5% |
| .net | 332 | 31 | 9.3% |
| .de (Germany) | 201 | 16 | 8% |
| .pl (Poland) | 52 | 4 | 7.7% |
| .cz (Czechia) | 32 | 2 | 6.3% |
| .nl (Netherlands) | 48 | 3 | 6.3% |
| .ch (Switzerland) | 35 | 2 | 5.7% |
Top 500 of 1,391 sites with an llms.txt
| # | Site | Category | Policy |
|---|---|---|---|
| 2 | cloudflare.com | Tech | Open |
| 15 | github.com | Tech | Blocks training only |
| 22 | wordpress.org | Tech | Open |
| 30 | adobe.com | Tech | Open |
| 37 | opera.com | Tech | Open |
| 41 | sentry.io | Tech | Open |
| 46 | samsung.com | Tech | Open |
| 57 | dropbox.com | Tech | Open |
| 58 | intuit.com | Finance | Open |
| 63 | kaspersky.com | Tech | Open |
| 70 | webex.com | Tech | Unreadable |
| 71 | paypal.com | Finance | Open |
| 72 | shopify.com | E-commerce | Open |
| 75 | base.org | Other | Open |
| 93 | ubuntu.com | Tech | Open |
| 100 | cisco.com | Tech | Open |
| 104 | stripe.com | Tech | Open |
| 112 | hubspot.com | Tech | Open |
| 113 | roku.com | Other | Open |
| 135 | weather.com | News & media | Blocks some AI search |
| 137 | dell.com | Tech | Open |
| 141 | etsy.com | E-commerce | Open |
| 143 | name.com | Other | Blocks training only |
| 146 | bilibili.com | Entertainment | Open |
| 149 | slack.com | Tech | Open |
| 152 | ea.com | Entertainment | Blocks training only |
| 153 | checkpoint.com | Tech | Open |
| 155 | selectel.ru | Other | Open |
| 163 | zendesk.com | Tech | Open |
| 164 | nvidia.com | Tech | Open |
| 171 | plesk.com | Tech | Open |
| 172 | calendly.com | Tech | Blocks training only |
| 186 | dailymotion.com | Entertainment | Blocks training only |
| 192 | intel.com | Tech | Open |
| 195 | paloaltonetworks.com | Tech | Open |
| 196 | substack.com | News & media | Open |
| 197 | bluehost.com | Tech | Unreadable |
| 199 | businessinsider.com | News & media | Blocks training only |
| 208 | okta.com | Tech | Open |
| 218 | shein.com | E-commerce | Open |
| 228 | foxnews.com | News & media | Open |
| 236 | bitrix24.ru | Other | Open |
| 237 | capcut.com | Other | Open |
| 239 | trendmicro.com | Tech | Open |
| 240 | squarespace.com | Tech | Open |
| 246 | digitalocean.com | Tech | Open |
| 258 | dreamhost.com | Tech | Open |
| 259 | shop.app | E-commerce | Open |
| 265 | playstation.com | Entertainment | Open |
| 266 | klaviyo.com | Tech | Open |
| 267 | target.com | E-commerce | Open |
| 272 | atlassian.com | Tech | Open |
| 278 | wix.com | Tech | Open |
| 290 | intercom.io | Tech | Open |
| 293 | redhat.com | Tech | Open |
| 299 | surveymonkey.com | Tech | Open |
| 303 | sophos.com | Tech | Blocks training only |
| 305 | porkbun.com | Other | Open |
| 306 | allegro.pl | E-commerce | Open |
| 313 | dailymail.com | News & media | Blocks some AI search |
| 318 | wps.com | Tech | Open |
| 319 | ring.com | Tech | Open |
| 321 | oxylabs.io | Other | Open |
| 330 | asus.com | Tech | Open |
| 333 | mailchimp.com | Tech | Open |
| 354 | fiverr.com | E-commerce | Open |
| 359 | prnewswire.com | News & media | Open |
| 364 | zoho.com | Tech | Open |
| 365 | cursor.sh | AI | Open |
| 379 | att.com | Telecom | Open |
| 385 | pendo.io | Other | Open |
| 390 | frontiersin.org | Reference | Open |
| 391 | cloudinary.com | Tech | Open |
| 403 | lg.com | Tech | Open |
| 406 | agora.io | Other | Open |
| 424 | tailscale.com | Tech | Open |
| 429 | expedia.com | Travel | Open |
| 430 | repubblica.it | News & media | Blocks some AI search |
| 431 | otto.de | E-commerce | Open |
| 446 | flightradar24.com | Entertainment | Blocks training only |
| 455 | jimdo.com | Tech | Open |
| 456 | xing.com | Social | Open |
| 473 | typeform.com | Tech | Open |
| 481 | supabase.co | Tech | Open |
| 482 | instructure.com | Tech | Open |
| 487 | netgear.com | Tech | Open |
| 491 | terra.com.br | News & media | Open |
| 503 | classlink.com | Tech | Open |
| 514 | lge.com | Other | Open |
| 523 | auth0.com | Tech | Open |
| 536 | one.com | Other | Open |
| 551 | clever.com | Tech | Open |
| 560 | siemens.com | Tech | Open |
| 563 | coursera.org | Tech | Blocks training only |
| 571 | marktplaats.nl | E-commerce | Open |
| 576 | savefrom.net | Tech | Open |
| 577 | amap.com | Travel | Blocks some AI search |
| 593 | jusbrasil.com.br | Other | Blocks some AI search |
| 622 | rajasthan.gov.in | Government | Open |
| 628 | edna.ru | Other | Open |
| 655 | haberler.com | News & media | Open |
| 658 | nesine.com | Finance | Open |
| 661 | bookmyshow.com | Other | Open |
| 673 | ilmeteo.it | News & media | Blocks training only |
| 687 | protothema.gr | News & media | Open |
| 690 | home.pl | Other | Open |
| 699 | sozcu.com.tr | News & media | Open |
| 710 | rustdesk.com | Tech | Blocks training only |
| 716 | markt.de | E-commerce | Open |
| 726 | sondakika.com | News & media | Open |
| 732 | utorrent.com | Tech | Open |
| 733 | notion.com | Other | Blocks some AI search |
| 738 | programme-tv.net | Other | Blocks training only |
| 741 | musicdel.ir | Entertainment | Open |
| 751 | rakuten-sec.co.jp | Other | Open |
| 761 | therapservices.net | Other | Open |
| 764 | wordwall.net | Other | Open |
| 768 | mongodb.com | Other | Open |
| 772 | chordtela.com | Other | Open |
| 781 | nustargame.com | Entertainment | Open |
| 790 | alphapolis.co.jp | Other | Open |
| 793 | wpastra.com | Other | Open |
| 803 | eff.org | Other | Open |
| 806 | arionplay.com | Entertainment | Open |
| 823 | moe.video | Entertainment | Open |
| 833 | sambad.com | Other | Open |
| 839 | hostinger.com | Tech | Blocks training only |
| 842 | verify.hn | Other | Open |
| 846 | akismet.com | Other | Open |
| 852 | xfinity.com | Telecom | Open |
| 863 | loopia.se | Other | Open |
| 869 | adroll.com | Other | Open |
| 873 | rapid7.com | Other | Open |
| 884 | pewresearch.org | Other | Open |
| 899 | gdemoideti.ru | Other | Open |
| 914 | dantri.com.vn | Other | Open |
| 915 | clevelandclinic.org | Other | Open |
| 942 | oreilly.com | Other | Blocks training only |
| 943 | iubenda.com | Other | Open |
| 948 | bigcommerce.com | Other | Open |
| 949 | moengage.com | Other | Open |
| 955 | clickpetroleoegas.com.br | Other | Open |
| 959 | alura.com.br | Other | Open |
| 960 | mindbox.ru | Other | Open |
| 969 | ninjarmm.com | Other | Open |
| 981 | squareup.com | Other | Open |
| 987 | wpengine.com | Other | Open |
| 994 | meetup.com | Social | Blocks training only |
| 997 | dondominio.com | Other | Open |
| 998 | tvmon1.com | Other | Open |
| 1003 | livechatinc.com | Other | Open |
| 1004 | privacymanager.io | Other | Open |
| 1011 | logitech.com | Other | Open |
| 1014 | intentiq.com | Other | Open |
| 1022 | getbootstrap.com | Other | Open |
| 1034 | acronis.com | Other | Open |
| 1052 | blacktoon418.com | Other | Open |
| 1059 | revenuecat.com | Other | Open |
| 1061 | evernote.com | Other | Open |
| 1066 | primis.tech | Other | Open |
| 1067 | yotpo.com | Other | Open |
| 1071 | trip.com | Other | Open |
| 1082 | fliphtml5.com | Other | Blocks training only |
| 1083 | shodan.io | Other | Open |
| 1086 | metamask.io | Other | Open |
| 1093 | datto.com | Other | Open |
| 1098 | posthog.com | News & media | Open |
| 1106 | adkernel.com | Other | No robots.txt |
| 1112 | nodejs.org | Tech | Open |
| 1118 | mgid.com | Other | Open |
| 1120 | asana.com | Other | Open |
| 1121 | semrush.com | Other | Open |
| 1122 | constantcontact.com | Other | Open |
| 1139 | elastic.co | Other | Open |
| 1143 | customer.io | Other | Open |
| 1149 | cookiebot.com | Other | Open |
| 1157 | innovid.com | Other | Open |
| 1158 | useinsider.com | Other | Open |
| 1161 | inmotionhosting.com | Other | Open |
| 1163 | websupport.sk | Other | Open |
| 1165 | msu.edu | Education | Open |
| 1167 | sendbird.com | Other | Blocks training only |
| 1171 | grafana.com | Other | Open |
| 1173 | screenconnect.com | Other | Open |
| 1179 | ucoz.ru | Other | Open |
| 1180 | fullstory.com | Other | Open |
| 1183 | simpli.fi | Other | Open |
| 1185 | verkada.com | Other | Open |
| 1188 | crowdstrike.com | Other | Open |
| 1200 | klarna.com | Other | Open |
| 1205 | bigcartel.com | Other | Open |
| 1212 | nielsen.com | Other | Open |
| 1213 | gameanalytics.com | Entertainment | Blocks training only |
| 1223 | ringcentral.com | Other | Open |
| 1226 | pubnative.net | Other | Open |
| 1241 | nordvpn.com | Other | Open |
| 1242 | surfshark.com | Other | Open |
| 1243 | ford.com | Other | Open |
| 1256 | okx.com | Other | Open |
| 1265 | here.com | Other | Open |
| 1266 | upwork.com | Other | Open |
| 1272 | coinmarketcap.com | Finance | Open |
| 1286 | canonical.com | Other | Open |
| 1291 | digg.com | Other | Open |
| 1292 | fotmob.com | Other | Open |
| 1294 | urbanairship.com | Other | Open |
| 1296 | snapcraft.io | Other | Open |
| 1307 | ifeng.com | Other | Open |
| 1309 | railway.app | Other | Open |
| 1350 | slashdot.org | Other | Open |
| 1358 | weplayapp.com | Entertainment | Open |
| 1363 | parallels.com | Other | Open |
| 1373 | databricks.com | Other | Open |
| 1382 | tenable.com | Other | Open |
| 1384 | clickup.com | Other | Open |
| 1385 | visualwebsiteoptimizer.com | Other | Open |
| 1391 | tutu.ru | Other | Open |
| 1392 | joinmassive.com | Other | Open |
| 1399 | cookieyes.com | Other | Open |
| 1401 | woocommerce.com | Other | Open |
| 1410 | padlet.com | Other | Open |
| 1415 | nsw.gov.au | Government | Open |
| 1420 | kochava.com | Other | Open |
| 1423 | vtex.com | Other | Open |
| 1425 | hindustantimes.com | News & media | Blocks some AI search |
| 1426 | ihc.ru | Other | Open |
| 1428 | sportradar.com | News & media | Open |
| 1429 | redis.io | Other | Open |
| 1430 | suno.com | Other | Open |
| 1451 | cutt.ly | Other | Open |
| 1458 | zscaler.com | Other | Open |
| 1467 | cloudx.io | Other | Open |
| 1469 | shazam.com | Other | Open |
| 1472 | cldin.eu | Other | Open |
| 1485 | gumroad.com | Other | Open |
| 1512 | tvsquared.com | Other | Open |
| 1516 | square.site | Other | Open |
| 1529 | prezi.com | Other | Blocks training only |
| 1531 | phys.org | Other | Open |
| 1532 | thawte.com | Other | Open |
| 1535 | zapier.com | Other | Open |
| 1553 | crazyegg.com | Other | Open |
| 1556 | robinhood.com | Other | Open |
| 1559 | crealitycloud.com | Other | Open |
| 1562 | laracasts.com | Other | Open |
| 1572 | kpmg.com | Other | Open |
| 1573 | kbz0pwvxmv.com | Other | Open |
| 1590 | malwarebytes.com | Other | Open |
| 1592 | libsyn.com | Other | Open |
| 1614 | vercel.com | Tech | Open |
| 1617 | goguardian.com | Other | Open |
| 1618 | rudderstack.com | Other | Open |
| 1623 | lovable.dev | Other | Open |
| 1626 | ex.co | Other | Open |
| 1628 | egnyte.com | Other | Open |
| 1637 | drpc.org | Other | Open |
| 1639 | duosecurity.com | Other | Open |
| 1640 | loom.com | Other | Blocks training only |
| 1645 | brevo.com | Other | Open |
| 1648 | threatlocker.com | Other | Open |
| 1654 | imagemagick.org | Other | Open |
| 1656 | xero.com | Other | Open |
| 1662 | newsbreak.com | News & media | Blocks training only |
| 1669 | mountain.com | Other | Open |
| 1671 | c6gj-static.net | Other | Open |
| 1678 | ip.sb | Other | Open |
| 1692 | guard.io | Other | Open |
| 1694 | yg5sjx5kzy.com | Other | Open |
| 1709 | sproutsocial.com | Other | Open |
| 1712 | monday.com | Other | Open |
| 1721 | walkme.com | Other | Open |
| 1727 | fool.com | Other | Blocks training only |
| 1737 | traveloka.com | Other | Open |
| 1742 | goodhousekeeping.com | Other | Blocks training only |
| 1750 | voximplant.com | Other | Open |
| 1759 | cvent.com | Other | Open |
| 1761 | iterable.com | Other | Open |
| 1765 | mediavine.com | Other | Open |
| 1770 | hootsuite.com | Other | Open |
| 1775 | razorpay.com | Finance | Blocks training only |
| 1783 | giantpanda.com | Other | Open |
| 1785 | honor.com | Other | Open |
| 1799 | revcontent.com | Other | Open |
| 1800 | poste.it | News & media | Open |
| 1804 | lavanguardia.com | Other | Blocks training only |
| 1814 | postman.com | News & media | Blocks training only |
| 1821 | e-recht24.de | Other | Open |
| 1844 | dinahosting.com | Other | Open |
| 1846 | hotmart.com | Other | Open |
| 1854 | jio.com | Other | Blocks training only |
| 1855 | mobilefuse.com | Other | Open |
| 1874 | qnap.com | Other | Open |
| 1878 | sardine.ai | Other | Open |
| 1881 | replit.com | Other | Open |
| 1887 | ably.io | Other | Open |
| 1893 | smartsheet.com | Other | Open |
| 1905 | edx.org | Other | Open |
| 1919 | zee5.com | Other | Open |
| 1921 | familysearch.org | Other | Open |
| 1922 | domaintools.com | Other | Open |
| 1925 | colorlib.com | Other | Open |
| 1935 | zenodo.org | Other | Open |
| 1940 | trimble.com | Other | Open |
| 1952 | osano.com | Other | Open |
| 1953 | redbull.com | Other | Open |
| 1957 | sprinklr.com | Other | Open |
| 1977 | garant.ru | Other | Open |
| 1990 | ctrip.com | Other | Open |
| 1994 | pushwoosh.com | Other | Open |
| 1999 | dashlane.com | Other | Open |
| 2003 | apkpure.com | Other | Open |
| 2008 | youm7.com | Other | Open |
| 2010 | dev.to | News & media | Open |
| 2013 | netlify.com | Tech | Open |
| 2014 | elementor.com | Other | Open |
| 2019 | backblaze.com | Other | Open |
| 2020 | elevenlabs.io | Other | Open |
| 2024 | jumpcloud.com | Other | Open |
| 2030 | themegrill.com | Other | Open |
| 2040 | rapidssl.com | Other | Open |
| 2046 | qualcomm.com | Other | Open |
| 2062 | piano.io | Other | Open |
| 2071 | arduino.cc | Other | Open |
| 2073 | wiki.gg | Other | Blocks some AI search |
| 2077 | adsbynimbus.com | Other | Open |
| 2084 | admaster.cc | Other | Open |
| 2093 | mysterium.network | Other | Open |
| 2099 | si.com | Other | Open |
| 2104 | demandbase.com | Other | Open |
| 2109 | razer.com | Other | Open |
| 2111 | goldapple.ru | Other | Open |
| 2114 | farfetch.com | Other | Open |
| 2121 | complianz.io | Other | Open |
| 2123 | gamma.app | Other | Open |
| 2129 | instabug.com | Other | Open |
| 2130 | fox.com | Other | Open |
| 2132 | xe.com | Other | Open |
| 2133 | proximus.be | Other | Open |
| 2134 | dmca.com | Other | Open |
| 2135 | bitwarden.com | Other | Open |
| 2140 | sketchup.com | Other | Open |
| 2142 | twingate.com | Other | Open |
| 2145 | cloud4box.ru | Other | Open |
| 2146 | similarweb.com | Other | Open |
| 2147 | jedox.com | Other | Open |
| 2151 | openrouter.ai | Other | Open |
| 2156 | capitaloneshopping.com | E-commerce | Open |
| 2159 | tum.de | Other | Open |
| 2164 | konghq.com | Other | Open |
| 2176 | 255md.com | Other | Blocks some AI search |
| 2184 | lokalise.com | Other | Open |
| 2198 | brandmetrics.com | Other | Open |
| 2204 | nazwa.pl | Other | Open |
| 2211 | helpshift.com | Other | Open |
| 2215 | 1und1.de | Other | Open |
| 2216 | elespanol.com | Other | Open |
| 2217 | g2.com | Other | Blocks training only |
| 2226 | grandviewresearch.com | Other | Open |
| 2227 | aruba.it | Other | Open |
| 2255 | veeam.com | Other | Open |
| 2257 | heroku.com | Tech | Open |
| 2262 | immutable.com | Other | Open |
| 2267 | signifyd.com | Other | Open |
| 2273 | seekingalpha.com | Other | Blocks some AI search |
| 2277 | ollama.com | Other | Open |
| 2284 | rokt.com | Other | Open |
| 2289 | a-ads.com | Other | Open |
| 2291 | rome2rio.com | Other | Open |
| 2297 | db.com | Other | Open |
| 2316 | farlightgames.com | Entertainment | No robots.txt |
| 2322 | growthbook.io | Other | Open |
| 2332 | bitget.com | Other | Open |
| 2339 | mux.com | Other | Open |
| 2341 | sanity.io | Other | Open |
| 2344 | apuestatotal.com | Other | Open |
| 2348 | appboy.com | Other | Open |
| 2351 | termly.io | Other | Open |
| 2355 | kia.com | Other | Open |
| 2356 | buffer.com | Other | Open |
| 2359 | adyen.com | Other | Open |
| 2360 | protectmyapp.com | Other | Open |
| 2366 | nanit.com | Other | Open |
| 2372 | mailerlite.com | Other | Open |
| 2378 | compass.com | Other | Open |
| 2405 | livemint.com | Other | Blocks some AI search |
| 2407 | neon.tech | Other | Open |
| 2412 | zoominfo.com | Other | Blocks some AI search |
| 2422 | greenhouse.io | Other | Open |
| 2429 | hyundai.com | Other | Open |
| 2432 | op.gg | Other | Open |
| 2435 | nationalarchives.gov.uk | Government | Open |
| 2437 | livechat.com | Other | Open |
| 2442 | ubs.com | Other | Open |
| 2478 | camscanner.com | Other | Open |
| 2484 | crypto.com | Finance | Open |
| 2491 | producthunt.com | Other | Open |
| 2494 | opensea.io | Other | Blocks training only |
| 2505 | paylocity.com | Finance | Open |
| 2510 | cyberfolks.pl | Other | Open |
| 2520 | higgsfield.ai | Other | Open |
| 2521 | terabox.com | Other | Open |
| 2522 | onamae.com | Other | Open |
| 2523 | adspower.net | Other | Open |
| 2528 | harica.gr | Other | No robots.txt |
| 2537 | victoriassecret.com | Other | Open |
| 2543 | tanium.com | Other | Open |
| 2556 | kaggle.com | Other | No robots.txt |
| 2560 | vedomosti.ru | Other | Open |
| 2571 | popsci.com | Other | Open |
| 2572 | searchenginejournal.com | News & media | Open |
| 2574 | kayak.com | Other | Open |
| 2577 | picsart.com | Other | Open |
| 2578 | actionverb.com | Other | Open |
| 2579 | preply.com | Other | Open |
| 2606 | thetrainline.com | Other | Open |
| 2614 | impact.com | Other | Blocks training only |
| 2616 | salesmartly.com | Other | Open |
| 2621 | newegg.com | Other | Open |
| 2631 | neilpatel.com | Other | Open |
| 2639 | krushmedia.com | Other | Open |
| 2650 | parentsquare.com | Other | Open |
| 2659 | expressvpn.com | News & media | Open |
| 2674 | turbify.com | Other | Blocks training only |
| 2683 | patch.com | Other | Blocks training only |
| 2691 | 1984.is | Other | Open |
| 2692 | amplify.com | Other | Blocks training only |
| 2701 | bankrate.com | Finance | Open |
| 2702 | bradesco.com.br | Other | Open |
| 2710 | coveo.com | Other | Open |
| 2714 | bittorrent.com | Other | Open |
| 2719 | eufylife.com | Other | Open |
| 2737 | rm358.com | Other | Blocks some AI search |
| 2749 | fpjs.io | Other | Open |
| 2754 | seaart.ai | Other | Open |
| 2755 | superbet.ro | Finance | Open |
| 2760 | stlouisfed.org | Other | Open |
| 2767 | msnbc.com | Other | Blocks some AI search |
| 2771 | 3ds.com | Other | Open |
| 2776 | magnite.com | Other | Open |
| 2786 | wedos.cz | Other | Open |
| 2788 | buildkite.com | Other | Open |
| 2797 | remotepc.com | Other | Open |
| 2798 | qualys.com | Other | Open |
| 2802 | glance.com | Other | Open |
| 2807 | lufthansa.com | Other | Open |
| 2809 | shb-sync.com | Other | Open |
| 2824 | findmykids.org | Other | Open |
| 2826 | motorola.com | Other | Open |
| 2833 | zoneedit.com | Other | No robots.txt |
| 2836 | infobip.com | Other | Blocks training only |
| 2841 | dynadot.com | Other | Open |
| 2860 | moovitapp.com | Other | Blocks training only |
| 2868 | theweathernetwork.com | News & media | Open |
| 2872 | airtel.in | Other | Open |
| 2876 | aap.org | Other | Open |
| 2883 | blackboard.com | Other | Open |
| 2884 | acehardware.com | Other | Open |
| 2907 | serverwatch.com | Other | Open |
| 2918 | pressreader.com | News & media | Open |
| 2926 | pixai.art | Other | Blocks training only |
| 2931 | wolfram.com | Other | Open |
| 2936 | ms.now | Other | Blocks some AI search |
| 2943 | ora.com | Other | Blocks training only |
| 2946 | remitly.com | Other | Open |
| 2963 | kijiji.ca | Other | Open |
| 2965 | getcomposer.org | Other | Open |
| 2968 | monster.com | Other | Open |
| 2974 | cricketwireless.com | Other | Open |
| 2987 | withings.net | Other | Open |
| 3002 | tinder.com | Other | Open |
| 3018 | outlookindia.com | Other | Open |
| 3025 | changeip.com | Other | Open |
| 3026 | bill.com | Other | Open |
| 3032 | uservoice.com | Other | Open |
| 3046 | techrepublic.com | Other | Open |
| 3052 | classdojo.com | Other | Open |
| 3053 | depositphotos.com | Other | Blocks training only |
| 3061 | skylinewebcams.com | Other | Open |
| 3062 | framer.com | Other | Open |
| 3102 | vuejs.org | Other | Open |
| 3105 | imperial.ac.uk | Education | Open |
| 3112 | abplive.com | Other | Open |
| 3113 | freeipapi.com | Other | Open |
| 3133 | nownodes.io | Other | Open |
| 3134 | wallpapers.com | Other | Open |
| 3144 | etherscan.io | Other | Open |
| 3148 | athemes.com | Other | Open |
| 3151 | ravelry.com | Other | Open |
| 3163 | thermofisher.com | Other | Open |
| 3166 | brivo.com | Other | Open |
| 3170 | milanuncios.com | Other | Blocks training only |
| 3177 | warwick.ac.uk | Education | Open |
| 3182 | imunify360.com | Other | Open |
| 3186 | vietnamnet.vn | Other | Open |
| 3187 | arkoselabs.com | Other | Blocks training only |
| 3192 | justgiving.com | Other | Open |
| 3212 | swarovski.com | Other | Open |
| 3219 | tradeinn.com | Other | Open |
| 3226 | survicate.com | Other | Open |
| 3230 | decodo.com | Other | Open |
Sorted by popularity. The full list with an llms_txt column is in the CSV.
FAQ
- What is llms.txt?
- llms.txt is a plain-text (Markdown) file at /llms.txt that gives AI models a short, curated guide to a website: what it is and which pages matter. It is a proposal, not a standard that every AI product reads.
- How many websites have an llms.txt?
- As of 2026-11-16, 1,391 of 8,313 popular websites (16.7%) serve an llms.txt file, the plain-text guide for AI models; among the top 100 sites it is 16 of 83. Sites with an llms.txt block GPTBot 5.9% of the time, sites without one 17.3%.
- Does llms.txt replace robots.txt?
- No. robots.txt tells crawlers what they may fetch; llms.txt suggests what to read. A site can serve both, and the data above shows how often they appear together.