AI Crawler Index · category
Do finance websites block AI crawlers?
As of 2026-11-16, 6.9% of the 102 finance websites whose robots.txt we could read block GPTBot, 6.9% block ClaudeBot and 4.9% block PerplexityBot (all sites: 15.4%, 14.7%, 8.2%). 23 of 150 publish an llms.txt file.
Block rate by crawler: finance vs all sites
| Crawler | Operator | Finance | All sites |
|---|---|---|---|
| CCBot | Common Crawl | 9.8% | 16.4% |
| Bytespider | ByteDance | 8.8% | 16.2% |
| Google-Extended | 7.8% | 13.2% | |
| GPTBot | OpenAI | 6.9% | 15.4% |
| ClaudeBot | Anthropic | 6.9% | 14.7% |
| Applebot-Extended | Apple | 6.9% | 13.4% |
| Meta-ExternalAgent | Meta | 6.9% | 13.6% |
| Amazonbot | Amazon | 6.9% | 13.2% |
| cohere-ai | Cohere | 6.9% | 12.7% |
| MistralAI-Training | Mistral | 5.9% | 7.1% |
| PerplexityBot | Perplexity | 4.9% | 8.2% |
| Google-CloudVertexBot | 4.9% | 8.8% | |
| OAI-AdsBot | OpenAI | 3.9% | 3% |
| Claude-SearchBot | Anthropic | 3.9% | 5.7% |
| Claude-User | Anthropic | 3.9% | 5.9% |
| Perplexity-User | Perplexity | 3.9% | 6.1% |
| Google-GeminiNotebook | 3.9% | 3.3% | |
| Google-Agent | 3.9% | 3.6% | |
| meta-webindexer | Meta | 3.9% | 5.1% |
| meta-externalfetcher | Meta | 3.9% | 6.5% |
| meta-externalads | Meta | 3.9% | 2.9% |
| Amzn-SearchBot | Amazon | 3.9% | 4.4% |
| Amzn-User | Amazon | 3.9% | 4.4% |
| DuckAssistBot | DuckDuckGo | 3.9% | 5.9% |
| MistralAI-User | Mistral | 3.9% | 5.7% |
| MistralAI-Index | Mistral | 3.9% | 3.6% |
| OAI-SearchBot | OpenAI | 2.9% | 5.8% |
| ChatGPT-User | OpenAI | 2.9% | 7.4% |
| Applebot | Apple | 2.9% | 3.2% |
All 150 finance sites in the index
✕ blocked · ◐ partially blocked (some paths) · ✓ allowed · ? robots.txt unreadable. "#" is the position in our popularity-ordered sample.
| # | Site | Policy | GPTBot | OAI-Search | ChatGPT-User | ClaudeBot | Claude-Search | Perplexity | Google-Ext. | CCBot | llms.txt |
|---|---|---|---|---|---|---|---|---|---|---|---|
| 58 | intuit.com | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | yes |
| 71 | paypal.com | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | yes |
| 78 | paytmfirstgames.com | Unreadable | unknown | unknown | unknown | unknown | unknown | unknown | unknown | unknown | – |
| 407 | bankofamerica.com | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 411 | binance.com | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 420 | fidelity.com | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 480 | wise.com | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 519 | alipay.com | No robots.txt | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 638 | funpay.com | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 639 | hollywoodbets.net | Unreadable | unknown | unknown | unknown | unknown | unknown | unknown | unknown | unknown | – |
| 654 | sportybet.com | No robots.txt | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 658 | nesine.com | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | yes |
| 664 | tgju.org | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 702 | betway.co.za | Unreadable | unknown | unknown | unknown | unknown | unknown | unknown | unknown | unknown | – |
| 709 | gcash.com | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 714 | klikbca.com | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 771 | betika.com | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 799 | betano.pe | Unreadable | unknown | unknown | unknown | unknown | unknown | unknown | unknown | unknown | – |
| 810 | betking.com | Unreadable | unknown | unknown | unknown | unknown | unknown | unknown | unknown | unknown | – |
| 816 | betpawa.ug | Open | partial | allowed | partial | partial | allowed | partial | partial | allowed | – |
| 825 | triunfobet.com | Unreadable | unknown | unknown | unknown | unknown | unknown | unknown | unknown | unknown | – |
| 893 | investing.com | Unreadable | unknown | unknown | unknown | unknown | unknown | unknown | unknown | unknown | – |
| 944 | forebet.com | Unreadable | unknown | unknown | unknown | unknown | unknown | unknown | unknown | unknown | – |
| 989 | winner.bet | Unreadable | unknown | unknown | unknown | unknown | unknown | unknown | unknown | unknown | – |
| 1110 | nubank.com.br | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 1111 | americanexpress.com | Unreadable | unknown | unknown | unknown | unknown | unknown | unknown | unknown | unknown | – |
| 1172 | coinbase.com | Unreadable | unknown | unknown | unknown | unknown | unknown | unknown | unknown | unknown | – |
| 1272 | coinmarketcap.com | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | yes |
| 1325 | betweendigital.com | No robots.txt | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 1445 | capitalone.com | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 1518 | schwab.com | No robots.txt | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 1574 | wellsfargo.com | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 1689 | dbankedge.cn | Unreadable | unknown | unknown | unknown | unknown | unknown | unknown | unknown | unknown | – |
| 1719 | chase.com | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 1775 | razorpay.com | Blocks training only | partial | partial | partial | partial | allowed | partial | partial | partial | yes |
| 1847 | coingecko.com | Open | allowed | allowed | allowed | allowed | allowed | allowed | partial | allowed | – |
| 1884 | paysafecard.com | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 2083 | banki.ru | No robots.txt | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 2158 | teacherspayteachers.com | Unreadable | unknown | unknown | unknown | unknown | unknown | unknown | unknown | unknown | – |
| 2170 | coindesk.com | Blocks some AI search | blocked | allowed | allowed | blocked | allowed | blocked | blocked | blocked | – |
| 2302 | betterstackdata.com | No robots.txt | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 2362 | betcity.ru | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 2472 | bet365.com | Unreadable | unknown | unknown | unknown | unknown | unknown | unknown | unknown | unknown | – |
| 2484 | crypto.com | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | yes |
| 2499 | paycomonline.net | Blocks all AI search | blocked | blocked | blocked | blocked | blocked | blocked | blocked | blocked | – |
| 2505 | paylocity.com | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | yes |
| 2684 | gazprombank.ru | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 2701 | bankrate.com | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | yes |
| 2727 | softbank.jp | No robots.txt | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 2755 | superbet.ro | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | yes |
| 3109 | sberbank-tele.com | No robots.txt | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 3270 | paypalobjects.com | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 3280 | afterpay.com | Blocks all AI search | blocked | blocked | blocked | blocked | blocked | blocked | blocked | blocked | – |
| 3313 | betterhelp.com | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | yes |
| 3318 | kucoin.com | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | yes |
| 3364 | lloydsbank.co.uk | No robots.txt | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 3493 | bancointer.com.br | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 3676 | payhip.com | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 3881 | paychex.com | Unreadable | unknown | unknown | unknown | unknown | unknown | unknown | unknown | unknown | – |
| 3917 | fon.bet | Unreadable | unknown | unknown | unknown | unknown | unknown | unknown | unknown | unknown | – |
| 4074 | bankier.pl | Blocks training only | blocked | allowed | allowed | allowed | allowed | allowed | blocked | blocked | – |
| 4080 | usbank.com | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | yes |
| 4098 | tranche.finance | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 4279 | mizuhobank.co.jp | Unreadable | unknown | unknown | unknown | unknown | unknown | unknown | unknown | unknown | – |
| 4334 | bet9ja.com | Unreadable | unknown | unknown | unknown | unknown | unknown | unknown | unknown | unknown | – |
| 4354 | bitcoin.org | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 4448 | paykeeper.ru | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 4500 | paymaya.com | Unreadable | unknown | unknown | unknown | unknown | unknown | unknown | unknown | unknown | – |
| 4743 | coinmaster.com | No robots.txt | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 4747 | betfair.com | Unreadable | unknown | unknown | unknown | unknown | unknown | unknown | unknown | unknown | – |
| 4784 | bethesda.net | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 4914 | yesplay.bet | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | yes |
| 5003 | paytm.com | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 5042 | goldbet.it | Unreadable | unknown | unknown | unknown | unknown | unknown | unknown | unknown | unknown | – |
| 5046 | bankbazaar.com | Blocks training only | allowed | allowed | allowed | allowed | allowed | allowed | allowed | blocked | yes |
| 5052 | premierbet.com | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 5170 | clickbank.net | Unreadable | unknown | unknown | unknown | unknown | unknown | unknown | unknown | unknown | – |
| 5323 | takbet.com | Unreadable | unknown | unknown | unknown | unknown | unknown | unknown | unknown | unknown | – |
| 5394 | bet88.ph | Unreadable | unknown | unknown | unknown | unknown | unknown | unknown | unknown | unknown | – |
| 5427 | jugabet.cl | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 5515 | betflag.it | Unreadable | unknown | unknown | unknown | unknown | unknown | unknown | unknown | unknown | – |
| 5586 | betmexico.mx | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 5632 | okbet.com | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | yes |
| 5637 | diabetes.org | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 5880 | crystalbet.com | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 5900 | ecuabet.com | Unreadable | unknown | unknown | unknown | unknown | unknown | unknown | unknown | unknown | – |
| 5920 | betting.co.zw | Unreadable | unknown | unknown | unknown | unknown | unknown | unknown | unknown | unknown | – |
| 5936 | investorgain.com | Blocks training only | allowed | allowed | allowed | allowed | allowed | allowed | allowed | blocked | – |
| 6009 | bplay.bet.ar | Unreadable | unknown | unknown | unknown | unknown | unknown | unknown | unknown | unknown | – |
| 6019 | unlockffbeta.com | No robots.txt | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 6041 | interbet.co.za | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 6077 | betclic.ci | Unreadable | unknown | unknown | unknown | unknown | unknown | unknown | unknown | unknown | – |
| 6102 | elephantbet.co.mz | Unreadable | unknown | unknown | unknown | unknown | unknown | unknown | unknown | unknown | – |
| 6192 | paymetrust.net | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 6230 | stipepay.com | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 6245 | bet30.host | Unreadable | unknown | unknown | unknown | unknown | unknown | unknown | unknown | unknown | – |
| 6260 | 8pay1.com | Unreadable | unknown | unknown | unknown | unknown | unknown | unknown | unknown | unknown | – |
| 6276 | metabet.tv | Unreadable | unknown | unknown | unknown | unknown | unknown | unknown | unknown | unknown | – |
| 6292 | betproexch.live | Blocks all AI search | blocked | blocked | blocked | blocked | blocked | blocked | blocked | blocked | – |
| 6303 | ebet.co.sz | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 6314 | citycenteronline.bet.ar | Unreadable | unknown | unknown | unknown | unknown | unknown | unknown | unknown | unknown | – |
| 6345 | castlebet.com.na | No robots.txt | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 6365 | cricbet99.black | No robots.txt | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 6543 | infinitepay.io | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | yes |
| 6672 | melbetegypt.com | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 6679 | betmgm.com | Unreadable | unknown | unknown | unknown | unknown | unknown | unknown | unknown | unknown | – |
| 6715 | payermax.com | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | yes |
| 6718 | moonpay.com | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | yes |
| 6820 | odibets.com | Blocks some AI search | blocked | allowed | allowed | blocked | allowed | allowed | blocked | blocked | – |
| 6881 | scotiabank.com | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 6950 | investors.com | Blocks some AI search | partial | partial | partial | blocked | blocked | blocked | blocked | blocked | – |
| 6953 | betsson.bet.ar | Unreadable | unknown | unknown | unknown | unknown | unknown | unknown | unknown | unknown | – |
| 7055 | sspay01.com | Unreadable | unknown | unknown | unknown | unknown | unknown | unknown | unknown | unknown | – |
| 7057 | betjili365.vip | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 7129 | bankofengland.co.uk | Unreadable | unknown | unknown | unknown | unknown | unknown | unknown | unknown | unknown | – |
| 7134 | 1jugarbet.com | Unreadable | unknown | unknown | unknown | unknown | unknown | unknown | unknown | unknown | – |
| 7196 | indiafinance.online | Unreadable | unknown | unknown | unknown | unknown | unknown | unknown | unknown | unknown | – |
| 7205 | khabar.bet | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 7207 | zplaybet.xyz | Unreadable | unknown | unknown | unknown | unknown | unknown | unknown | unknown | unknown | – |
| 7233 | wechatpay.cn | No robots.txt | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 7441 | citibank.com | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 7459 | rabobank.nl | Unreadable | unknown | unknown | unknown | unknown | unknown | unknown | unknown | unknown | – |
| 7525 | nitropay.com | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 7594 | commbank.com.au | Open | partial | partial | partial | partial | partial | partial | partial | partial | yes |
| 7608 | cloudpayments.ru | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 7628 | visitorbet-resmi.com | Unreadable | unknown | unknown | unknown | unknown | unknown | unknown | unknown | unknown | – |
| 7687 | juga-bet-col.com | No robots.txt | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 7713 | betterttv.net | No robots.txt | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 7966 | picpay.com | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | yes |
| 7975 | profinance.ru | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 8033 | databank.com | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | yes |
| 8104 | corporatefinanceinstitute.com | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 8291 | skybet.com | Unreadable | unknown | unknown | unknown | unknown | unknown | unknown | unknown | unknown | – |
| 8298 | shopeepay.com.br | No robots.txt | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 8359 | betboom.ru | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 8476 | payu.com | No robots.txt | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 8489 | payscale.com | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 8788 | payoneer.com | Blocks some AI search | blocked | allowed | allowed | blocked | allowed | allowed | blocked | blocked | yes |
| 8873 | paycor.com | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | yes |
| 9034 | gobankingrates.com | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 9116 | mrbetlogin.com | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 9226 | otpbank.ru | Unreadable | unknown | unknown | unknown | unknown | unknown | unknown | unknown | unknown | – |
| 9364 | icicibank.com | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 9473 | coinglass.com | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 9600 | kasikornbank.com | Unreadable | unknown | unknown | unknown | unknown | unknown | unknown | unknown | unknown | – |
| 9674 | cryptopro.ru | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 9726 | bitcointalk.org | Unreadable | unknown | unknown | unknown | unknown | unknown | unknown | unknown | unknown | – |
| 9875 | mrbetgames.com | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 9889 | betonline.ag | Unreadable | unknown | unknown | unknown | unknown | unknown | unknown | unknown | unknown | – |
| 9959 | coingraph-update.com | No robots.txt | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
FAQ
- Do finance websites block AI crawlers?
- As of 2026-11-16, 6.9% of the 102 finance websites whose robots.txt we could read block GPTBot, 6.9% block ClaudeBot and 4.9% block PerplexityBot (all sites: 15.4%, 14.7%, 8.2%). 23 of 150 publish an llms.txt file.
- Which finance websites block GPTBot?
- coindesk.com, paycomonline.net, afterpay.com, bankier.pl, betproexch.live, odibets.com, payoneer.com.
Other categories
- AI GPTBot 10%
- E-commerce GPTBot 14.7%
- Education GPTBot 5.6%
- Entertainment GPTBot 19.6%
- Government GPTBot 3%
- News & media GPTBot 33.7%
- Other GPTBot 15.1%
- Reference GPTBot 27.9%
- Social GPTBot 37.3%
- Tech GPTBot 3.7%
- Telecom GPTBot 0%
- Travel GPTBot 15.4%