AI Crawler Index · category
Do government websites block AI crawlers?
As of 2026-11-16, 3% of the 203 government websites whose robots.txt we could read block GPTBot, 3% block ClaudeBot and 3% block PerplexityBot (all sites: 15.4%, 14.7%, 8.2%). 8 of 330 publish an llms.txt file.
Block rate by crawler: government vs all sites
| Crawler | Operator | Government | All sites |
|---|---|---|---|
| ChatGPT-User | OpenAI | 3.9% | 7.4% |
| Claude-User | Anthropic | 3.4% | 5.9% |
| Perplexity-User | Perplexity | 3.4% | 6.1% |
| meta-externalfetcher | Meta | 3.4% | 6.5% |
| MistralAI-User | Mistral | 3.4% | 5.7% |
| GPTBot | OpenAI | 3% | 15.4% |
| ClaudeBot | Anthropic | 3% | 14.7% |
| PerplexityBot | Perplexity | 3% | 8.2% |
| Google-GeminiNotebook | 3% | 3.3% | |
| Google-Agent | 3% | 3.6% | |
| Amazonbot | Amazon | 3% | 13.2% |
| Amzn-User | Amazon | 3% | 4.4% |
| CCBot | Common Crawl | 3% | 16.4% |
| Bytespider | ByteDance | 3% | 16.2% |
| Meta-ExternalAgent | Meta | 2.5% | 13.6% |
| cohere-ai | Cohere | 2.5% | 12.7% |
| OAI-SearchBot | OpenAI | 2% | 5.8% |
| Claude-SearchBot | Anthropic | 2% | 5.7% |
| Google-Extended | 2% | 13.2% | |
| Google-CloudVertexBot | 2% | 8.8% | |
| Applebot-Extended | Apple | 2% | 13.4% |
| meta-webindexer | Meta | 2% | 5.1% |
| DuckAssistBot | DuckDuckGo | 2% | 5.9% |
| OAI-AdsBot | OpenAI | 1.5% | 3% |
| Applebot | Apple | 1.5% | 3.2% |
| meta-externalads | Meta | 1.5% | 2.9% |
| Amzn-SearchBot | Amazon | 1.5% | 4.4% |
| MistralAI-Index | Mistral | 1.5% | 3.6% |
| MistralAI-Training | Mistral | 1.5% | 7.1% |
Top 150 of 330 government sites in the index
✕ blocked · ◐ partially blocked (some paths) · ✓ allowed · ? robots.txt unreadable. "#" is the position in our popularity-ordered sample.
| # | Site | Policy | GPTBot | OAI-Search | ChatGPT-User | ClaudeBot | Claude-Search | Perplexity | Google-Ext. | CCBot | llms.txt |
|---|---|---|---|---|---|---|---|---|---|---|---|
| 51 | europa.eu | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 60 | nih.gov | Unreadable | unknown | unknown | unknown | unknown | unknown | unknown | unknown | unknown | – |
| 88 | miit.gov.cn | Unreadable | unknown | unknown | unknown | unknown | unknown | unknown | unknown | unknown | – |
| 105 | nist.gov | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 127 | www.gov.uk | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 154 | cdc.gov | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 156 | un.org | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 162 | nasa.gov | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 183 | beian.gov.cn | Unreadable | unknown | unknown | unknown | unknown | unknown | unknown | unknown | unknown | – |
| 230 | noaa.gov | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 263 | ca.gov | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 275 | unesco.org | No robots.txt | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 289 | loc.gov | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 292 | service.gov.uk | No robots.txt | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 338 | fda.gov | No robots.txt | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 374 | usda.gov | Unreadable | unknown | unknown | unknown | unknown | unknown | unknown | unknown | unknown | – |
| 402 | worldbank.org | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 413 | epa.gov | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 419 | state.gov | Unreadable | unknown | unknown | unknown | unknown | unknown | unknown | unknown | unknown | – |
| 435 | caixa.gov.br | Unreadable | unknown | unknown | unknown | unknown | unknown | unknown | unknown | unknown | – |
| 453 | weforum.org | Unreadable | unknown | unknown | unknown | unknown | unknown | unknown | unknown | unknown | – |
| 465 | whitehouse.gov | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 468 | bund.de | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 485 | maricopa.gov | Open | partial | partial | allowed | partial | partial | partial | allowed | partial | – |
| 492 | oecd.org | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 564 | acesso.gov.br | No robots.txt | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 587 | canada.ca | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 590 | admin.ch | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 622 | rajasthan.gov.in | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | yes |
| 637 | gst.gov.in | Unreadable | unknown | unknown | unknown | unknown | unknown | unknown | unknown | unknown | – |
| 645 | irs.gov | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 651 | turkiye.gov.tr | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 669 | incometax.gov.in | Unreadable | unknown | unknown | unknown | unknown | unknown | unknown | unknown | unknown | – |
| 681 | uidai.gov.in | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 682 | inps.it | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 715 | epfindia.gov.in | Unreadable | unknown | unknown | unknown | unknown | unknown | unknown | unknown | unknown | – |
| 717 | nfse.gov.br | Unreadable | unknown | unknown | unknown | unknown | unknown | unknown | unknown | unknown | – |
| 720 | eci.gov.in | No robots.txt | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 746 | indianrail.gov.in | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 750 | parivahan.gov.in | No robots.txt | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 753 | afip.gob.ar | Unreadable | unknown | unknown | unknown | unknown | unknown | unknown | unknown | unknown | – |
| 755 | ftc.gov | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 759 | osym.gov.tr | Unreadable | unknown | unknown | unknown | unknown | unknown | unknown | unknown | unknown | – |
| 784 | jra.go.jp | Unreadable | unknown | unknown | unknown | unknown | unknown | unknown | unknown | unknown | – |
| 800 | kemkes.go.id | No robots.txt | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 811 | digitallocker.gov.in | Unreadable | unknown | unknown | unknown | unknown | unknown | unknown | unknown | unknown | – |
| 814 | dord.gov.in | Unreadable | unknown | unknown | unknown | unknown | unknown | unknown | unknown | unknown | – |
| 829 | cbl.gov.ly | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 878 | va.gov | Blocks training only | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 892 | nps.gov | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 923 | meb.gov.tr | Unreadable | unknown | unknown | unknown | unknown | unknown | unknown | unknown | unknown | – |
| 928 | usgs.gov | Unreadable | unknown | unknown | unknown | unknown | unknown | unknown | unknown | unknown | – |
| 967 | www.gob.mx | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 980 | sec.gov | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 1012 | ed.gov | Unreadable | unknown | unknown | unknown | unknown | unknown | unknown | unknown | unknown | – |
| 1031 | bls.gov | No robots.txt | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 1065 | hhs.gov | Unreadable | unknown | unknown | unknown | unknown | unknown | unknown | unknown | unknown | – |
| 1090 | house.gov | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 1125 | justice.gov | Blocks some AI search | allowed | allowed | blocked | allowed | allowed | allowed | allowed | allowed | – |
| 1197 | energy.gov | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 1216 | congress.gov | Blocks some AI search | blocked | blocked | blocked | blocked | blocked | blocked | blocked | blocked | – |
| 1220 | ny.gov | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 1249 | senate.gov | Unreadable | unknown | unknown | unknown | unknown | unknown | unknown | unknown | unknown | – |
| 1297 | ssa.gov | Unreadable | unknown | unknown | unknown | unknown | unknown | unknown | unknown | unknown | – |
| 1359 | dhs.gov | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 1402 | archives.gov | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 1415 | nsw.gov.au | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | yes |
| 1421 | medlineplus.gov | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 1463 | faa.gov | Unreadable | unknown | unknown | unknown | unknown | unknown | unknown | unknown | unknown | – |
| 1479 | treasury.gov | Unreadable | unknown | unknown | unknown | unknown | unknown | unknown | unknown | unknown | – |
| 1488 | dot.gov | Unreadable | unknown | unknown | unknown | unknown | unknown | unknown | unknown | unknown | – |
| 1492 | army.mil | Unreadable | unknown | unknown | unknown | unknown | unknown | unknown | unknown | unknown | – |
| 1494 | fcc.gov | Unreadable | unknown | unknown | unknown | unknown | unknown | unknown | unknown | unknown | – |
| 1500 | texas.gov | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 1509 | wa.gov | Blocks all AI search | blocked | blocked | blocked | blocked | blocked | blocked | blocked | blocked | – |
| 1513 | privacyshield.gov | Unreadable | unknown | unknown | unknown | unknown | unknown | unknown | unknown | unknown | – |
| 1527 | usa.gov | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 1607 | navy.mil | Unreadable | unknown | unknown | unknown | unknown | unknown | unknown | unknown | unknown | – |
| 1642 | uspto.gov | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 1661 | fbi.gov | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 1664 | cancer.gov | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 1680 | nyc.gov | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 1685 | dol.gov | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 1776 | federalregister.gov | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 1784 | usembassy.gov | Unreadable | unknown | unknown | unknown | unknown | unknown | unknown | unknown | unknown | – |
| 1822 | dataprivacyframework.gov | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 1863 | mhlw.go.jp | Unreadable | unknown | unknown | unknown | unknown | unknown | unknown | unknown | unknown | – |
| 1904 | legislation.gov.uk | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 1931 | osha.gov | Unreadable | unknown | unknown | unknown | unknown | unknown | unknown | unknown | unknown | – |
| 1956 | cisa.gov | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 1972 | sba.gov | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 1996 | clinicaltrials.gov | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 2105 | cia.gov | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 2117 | uscourts.gov | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 2120 | federalreserve.gov | No robots.txt | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 2162 | uscis.gov | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 2235 | tfl.gov.uk | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 2240 | deped.gov.ph | Unreadable | unknown | unknown | unknown | unknown | unknown | unknown | unknown | unknown | – |
| 2243 | eia.gov | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 2321 | cms.gov | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 2354 | administracionelectronica.gob.es | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 2369 | wcd.gov.in | Unreadable | unknown | unknown | unknown | unknown | unknown | unknown | unknown | unknown | – |
| 2383 | serpro.gov.br | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 2387 | bisp.gov.pk | Unreadable | unknown | unknown | unknown | unknown | unknown | unknown | unknown | unknown | – |
| 2435 | nationalarchives.gov.uk | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | yes |
| 2462 | af.mil | Unreadable | unknown | unknown | unknown | unknown | unknown | unknown | unknown | unknown | – |
| 2482 | jst.go.jp | Unreadable | unknown | unknown | unknown | unknown | unknown | unknown | unknown | unknown | – |
| 2501 | defense.gov | Unreadable | unknown | unknown | unknown | unknown | unknown | unknown | unknown | unknown | – |
| 2598 | nsf.gov | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 2612 | pa.gov | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 2617 | mo.gov | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 2715 | jma.go.jp | Unreadable | unknown | unknown | unknown | unknown | unknown | unknown | unknown | unknown | – |
| 2756 | nj.gov | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 2839 | govinfo.gov | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 2858 | nhtsa.gov | Unreadable | unknown | unknown | unknown | unknown | unknown | unknown | unknown | unknown | – |
| 2861 | cbp.gov | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 2927 | mass.gov | Unreadable | unknown | unknown | unknown | unknown | unknown | unknown | unknown | unknown | – |
| 2975 | health.mil | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 2977 | michigan.gov | Unreadable | unknown | unknown | unknown | unknown | unknown | unknown | unknown | unknown | – |
| 2993 | ons.gov.uk | No robots.txt | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 3073 | ecfr.gov | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 3129 | lbl.gov | Unreadable | unknown | unknown | unknown | unknown | unknown | unknown | unknown | unknown | – |
| 3135 | legifrance.gouv.fr | Unreadable | unknown | unknown | unknown | unknown | unknown | unknown | unknown | unknown | – |
| 3137 | nla.gov.au | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 3228 | meti.go.jp | Unreadable | unknown | unknown | unknown | unknown | unknown | unknown | unknown | unknown | – |
| 3254 | maryland.gov | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | yes |
| 3312 | copyright.gov | No robots.txt | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 3315 | agenziaentrate.gov.it | Unreadable | unknown | unknown | unknown | unknown | unknown | unknown | unknown | unknown | – |
| 3355 | samhsa.gov | Unreadable | unknown | unknown | unknown | unknown | unknown | unknown | unknown | unknown | – |
| 3391 | mlit.go.jp | Unreadable | unknown | unknown | unknown | unknown | unknown | unknown | unknown | unknown | – |
| 3422 | chinatax.gov.cn | Unreadable | unknown | unknown | unknown | unknown | unknown | unknown | unknown | unknown | – |
| 3424 | gao.gov | Blocks some AI search | allowed | allowed | allowed | allowed | allowed | blocked | allowed | allowed | – |
| 3432 | kerala.gov.in | No robots.txt | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 3458 | tn.gov | Unreadable | unknown | unknown | unknown | unknown | unknown | unknown | unknown | unknown | – |
| 3464 | argentina.gob.ar | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 3554 | oregon.gov | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 3620 | radio-canada.ca | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 3679 | interno.gov.it | Unreadable | unknown | unknown | unknown | unknown | unknown | unknown | unknown | unknown | – |
| 3681 | supremecourt.gov | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 3697 | hud.gov | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 3738 | gpo.gov | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 3745 | gsa.gov | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 3802 | iowa.gov | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 3830 | consumerfinance.gov | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 3889 | login.gov | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 3933 | fws.gov | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 3965 | sunat.gob.pe | Unreadable | unknown | unknown | unknown | unknown | unknown | unknown | unknown | unknown | – |
| 4089 | mofcom.gov.cn | Unreadable | unknown | unknown | unknown | unknown | unknown | unknown | unknown | unknown | – |
| 4101 | sat.gob.mx | Unreadable | unknown | unknown | unknown | unknown | unknown | unknown | unknown | unknown | – |
| 4162 | mohfw.gov.in | Unreadable | unknown | unknown | unknown | unknown | unknown | unknown | unknown | unknown | – |
Sorted by popularity. Every site has its own page: all 10,000 sites (A-Z lists), or download the CSV.
FAQ
- Do government websites block AI crawlers?
- As of 2026-11-16, 3% of the 203 government websites whose robots.txt we could read block GPTBot, 3% block ClaudeBot and 3% block PerplexityBot (all sites: 15.4%, 14.7%, 8.2%). 8 of 330 publish an llms.txt file.
- Which government websites block GPTBot?
- congress.gov, wa.gov, nv.gov, educationboardresults.gov.bd, ahrq.gov, vermont.gov.
Other categories
- AI GPTBot 10%
- E-commerce GPTBot 14.7%
- Education GPTBot 5.6%
- Entertainment GPTBot 19.6%
- Finance GPTBot 6.9%
- News & media GPTBot 33.7%
- Other GPTBot 15.1%
- Reference GPTBot 27.9%
- Social GPTBot 37.3%
- Tech GPTBot 3.7%
- Telecom GPTBot 0%
- Travel GPTBot 15.4%