TidyToolsAI Crawler Index · data as of · updated daily

AI Crawler Index · category

Do finance websites block AI crawlers?

As of 2026-11-16, 6.9% of the 102 finance websites whose robots.txt we could read block GPTBot, 6.9% block ClaudeBot and 4.9% block PerplexityBot (all sites: 15.4%, 14.7%, 8.2%). 23 of 150 publish an llms.txt file.

Block rate by crawler: finance vs all sites

CrawlerOperatorFinanceAll sites
CCBotCommon Crawl9.8%16.4%
BytespiderByteDance8.8%16.2%
Google-ExtendedGoogle7.8%13.2%
GPTBotOpenAI6.9%15.4%
ClaudeBotAnthropic6.9%14.7%
Applebot-ExtendedApple6.9%13.4%
Meta-ExternalAgentMeta6.9%13.6%
AmazonbotAmazon6.9%13.2%
cohere-aiCohere6.9%12.7%
MistralAI-TrainingMistral5.9%7.1%
PerplexityBotPerplexity4.9%8.2%
Google-CloudVertexBotGoogle4.9%8.8%
OAI-AdsBotOpenAI3.9%3%
Claude-SearchBotAnthropic3.9%5.7%
Claude-UserAnthropic3.9%5.9%
Perplexity-UserPerplexity3.9%6.1%
Google-GeminiNotebookGoogle3.9%3.3%
Google-AgentGoogle3.9%3.6%
meta-webindexerMeta3.9%5.1%
meta-externalfetcherMeta3.9%6.5%
meta-externaladsMeta3.9%2.9%
Amzn-SearchBotAmazon3.9%4.4%
Amzn-UserAmazon3.9%4.4%
DuckAssistBotDuckDuckGo3.9%5.9%
MistralAI-UserMistral3.9%5.7%
MistralAI-IndexMistral3.9%3.6%
OAI-SearchBotOpenAI2.9%5.8%
ChatGPT-UserOpenAI2.9%7.4%
ApplebotApple2.9%3.2%

All 150 finance sites in the index

✕ blocked · ◐ partially blocked (some paths) · ✓ allowed · ? robots.txt unreadable. "#" is the position in our popularity-ordered sample.

#SitePolicyGPTBotOAI-SearchChatGPT-UserClaudeBotClaude-SearchPerplexityGoogle-Ext.CCBotllms.txt
58intuit.comOpenallowedallowedallowedallowedallowedallowedallowedallowedyes
71paypal.comOpenallowedallowedallowedallowedallowedallowedallowedallowedyes
78paytmfirstgames.comUnreadableunknownunknownunknownunknownunknownunknownunknownunknown–
407bankofamerica.comOpenallowedallowedallowedallowedallowedallowedallowedallowed–
411binance.comOpenallowedallowedallowedallowedallowedallowedallowedallowed–
420fidelity.comOpenallowedallowedallowedallowedallowedallowedallowedallowed–
480wise.comOpenallowedallowedallowedallowedallowedallowedallowedallowed–
519alipay.comNo robots.txtallowedallowedallowedallowedallowedallowedallowedallowed–
638funpay.comOpenallowedallowedallowedallowedallowedallowedallowedallowed–
639hollywoodbets.netUnreadableunknownunknownunknownunknownunknownunknownunknownunknown–
654sportybet.comNo robots.txtallowedallowedallowedallowedallowedallowedallowedallowed–
658nesine.comOpenallowedallowedallowedallowedallowedallowedallowedallowedyes
664tgju.orgOpenallowedallowedallowedallowedallowedallowedallowedallowed–
702betway.co.zaUnreadableunknownunknownunknownunknownunknownunknownunknownunknown–
709gcash.comOpenallowedallowedallowedallowedallowedallowedallowedallowed–
714klikbca.comOpenallowedallowedallowedallowedallowedallowedallowedallowed–
771betika.comOpenallowedallowedallowedallowedallowedallowedallowedallowed–
799betano.peUnreadableunknownunknownunknownunknownunknownunknownunknownunknown–
810betking.comUnreadableunknownunknownunknownunknownunknownunknownunknownunknown–
816betpawa.ugOpenpartialallowedpartialpartialallowedpartialpartialallowed–
825triunfobet.comUnreadableunknownunknownunknownunknownunknownunknownunknownunknown–
893investing.comUnreadableunknownunknownunknownunknownunknownunknownunknownunknown–
944forebet.comUnreadableunknownunknownunknownunknownunknownunknownunknownunknown–
989winner.betUnreadableunknownunknownunknownunknownunknownunknownunknownunknown–
1110nubank.com.brOpenallowedallowedallowedallowedallowedallowedallowedallowed–
1111americanexpress.comUnreadableunknownunknownunknownunknownunknownunknownunknownunknown–
1172coinbase.comUnreadableunknownunknownunknownunknownunknownunknownunknownunknown–
1272coinmarketcap.comOpenallowedallowedallowedallowedallowedallowedallowedallowedyes
1325betweendigital.comNo robots.txtallowedallowedallowedallowedallowedallowedallowedallowed–
1445capitalone.comOpenallowedallowedallowedallowedallowedallowedallowedallowed–
1518schwab.comNo robots.txtallowedallowedallowedallowedallowedallowedallowedallowed–
1574wellsfargo.comOpenallowedallowedallowedallowedallowedallowedallowedallowed–
1689dbankedge.cnUnreadableunknownunknownunknownunknownunknownunknownunknownunknown–
1719chase.comOpenallowedallowedallowedallowedallowedallowedallowedallowed–
1775razorpay.comBlocks training onlypartialpartialpartialpartialallowedpartialpartialpartialyes
1847coingecko.comOpenallowedallowedallowedallowedallowedallowedpartialallowed–
1884paysafecard.comOpenallowedallowedallowedallowedallowedallowedallowedallowed–
2083banki.ruNo robots.txtallowedallowedallowedallowedallowedallowedallowedallowed–
2158teacherspayteachers.comUnreadableunknownunknownunknownunknownunknownunknownunknownunknown–
2170coindesk.comBlocks some AI searchblockedallowedallowedblockedallowedblockedblockedblocked–
2302betterstackdata.comNo robots.txtallowedallowedallowedallowedallowedallowedallowedallowed–
2362betcity.ruOpenallowedallowedallowedallowedallowedallowedallowedallowed–
2472bet365.comUnreadableunknownunknownunknownunknownunknownunknownunknownunknown–
2484crypto.comOpenallowedallowedallowedallowedallowedallowedallowedallowedyes
2499paycomonline.netBlocks all AI searchblockedblockedblockedblockedblockedblockedblockedblocked–
2505paylocity.comOpenallowedallowedallowedallowedallowedallowedallowedallowedyes
2684gazprombank.ruOpenallowedallowedallowedallowedallowedallowedallowedallowed–
2701bankrate.comOpenallowedallowedallowedallowedallowedallowedallowedallowedyes
2727softbank.jpNo robots.txtallowedallowedallowedallowedallowedallowedallowedallowed–
2755superbet.roOpenallowedallowedallowedallowedallowedallowedallowedallowedyes
3109sberbank-tele.comNo robots.txtallowedallowedallowedallowedallowedallowedallowedallowed–
3270paypalobjects.comOpenallowedallowedallowedallowedallowedallowedallowedallowed–
3280afterpay.comBlocks all AI searchblockedblockedblockedblockedblockedblockedblockedblocked–
3313betterhelp.comOpenallowedallowedallowedallowedallowedallowedallowedallowedyes
3318kucoin.comOpenallowedallowedallowedallowedallowedallowedallowedallowedyes
3364lloydsbank.co.ukNo robots.txtallowedallowedallowedallowedallowedallowedallowedallowed–
3493bancointer.com.brOpenallowedallowedallowedallowedallowedallowedallowedallowed–
3676payhip.comOpenallowedallowedallowedallowedallowedallowedallowedallowed–
3881paychex.comUnreadableunknownunknownunknownunknownunknownunknownunknownunknown–
3917fon.betUnreadableunknownunknownunknownunknownunknownunknownunknownunknown–
4074bankier.plBlocks training onlyblockedallowedallowedallowedallowedallowedblockedblocked–
4080usbank.comOpenallowedallowedallowedallowedallowedallowedallowedallowedyes
4098tranche.financeOpenallowedallowedallowedallowedallowedallowedallowedallowed–
4279mizuhobank.co.jpUnreadableunknownunknownunknownunknownunknownunknownunknownunknown–
4334bet9ja.comUnreadableunknownunknownunknownunknownunknownunknownunknownunknown–
4354bitcoin.orgOpenallowedallowedallowedallowedallowedallowedallowedallowed–
4448paykeeper.ruOpenallowedallowedallowedallowedallowedallowedallowedallowed–
4500paymaya.comUnreadableunknownunknownunknownunknownunknownunknownunknownunknown–
4743coinmaster.comNo robots.txtallowedallowedallowedallowedallowedallowedallowedallowed–
4747betfair.comUnreadableunknownunknownunknownunknownunknownunknownunknownunknown–
4784bethesda.netOpenallowedallowedallowedallowedallowedallowedallowedallowed–
4914yesplay.betOpenallowedallowedallowedallowedallowedallowedallowedallowedyes
5003paytm.comOpenallowedallowedallowedallowedallowedallowedallowedallowed–
5042goldbet.itUnreadableunknownunknownunknownunknownunknownunknownunknownunknown–
5046bankbazaar.comBlocks training onlyallowedallowedallowedallowedallowedallowedallowedblockedyes
5052premierbet.comOpenallowedallowedallowedallowedallowedallowedallowedallowed–
5170clickbank.netUnreadableunknownunknownunknownunknownunknownunknownunknownunknown–
5323takbet.comUnreadableunknownunknownunknownunknownunknownunknownunknownunknown–
5394bet88.phUnreadableunknownunknownunknownunknownunknownunknownunknownunknown–
5427jugabet.clOpenallowedallowedallowedallowedallowedallowedallowedallowed–
5515betflag.itUnreadableunknownunknownunknownunknownunknownunknownunknownunknown–
5586betmexico.mxOpenallowedallowedallowedallowedallowedallowedallowedallowed–
5632okbet.comOpenallowedallowedallowedallowedallowedallowedallowedallowedyes
5637diabetes.orgOpenallowedallowedallowedallowedallowedallowedallowedallowed–
5880crystalbet.comOpenallowedallowedallowedallowedallowedallowedallowedallowed–
5900ecuabet.comUnreadableunknownunknownunknownunknownunknownunknownunknownunknown–
5920betting.co.zwUnreadableunknownunknownunknownunknownunknownunknownunknownunknown–
5936investorgain.comBlocks training onlyallowedallowedallowedallowedallowedallowedallowedblocked–
6009bplay.bet.arUnreadableunknownunknownunknownunknownunknownunknownunknownunknown–
6019unlockffbeta.comNo robots.txtallowedallowedallowedallowedallowedallowedallowedallowed–
6041interbet.co.zaOpenallowedallowedallowedallowedallowedallowedallowedallowed–
6077betclic.ciUnreadableunknownunknownunknownunknownunknownunknownunknownunknown–
6102elephantbet.co.mzUnreadableunknownunknownunknownunknownunknownunknownunknownunknown–
6192paymetrust.netOpenallowedallowedallowedallowedallowedallowedallowedallowed–
6230stipepay.comOpenallowedallowedallowedallowedallowedallowedallowedallowed–
6245bet30.hostUnreadableunknownunknownunknownunknownunknownunknownunknownunknown–
62608pay1.comUnreadableunknownunknownunknownunknownunknownunknownunknownunknown–
6276metabet.tvUnreadableunknownunknownunknownunknownunknownunknownunknownunknown–
6292betproexch.liveBlocks all AI searchblockedblockedblockedblockedblockedblockedblockedblocked–
6303ebet.co.szOpenallowedallowedallowedallowedallowedallowedallowedallowed–
6314citycenteronline.bet.arUnreadableunknownunknownunknownunknownunknownunknownunknownunknown–
6345castlebet.com.naNo robots.txtallowedallowedallowedallowedallowedallowedallowedallowed–
6365cricbet99.blackNo robots.txtallowedallowedallowedallowedallowedallowedallowedallowed–
6543infinitepay.ioOpenallowedallowedallowedallowedallowedallowedallowedallowedyes
6672melbetegypt.comOpenallowedallowedallowedallowedallowedallowedallowedallowed–
6679betmgm.comUnreadableunknownunknownunknownunknownunknownunknownunknownunknown–
6715payermax.comOpenallowedallowedallowedallowedallowedallowedallowedallowedyes
6718moonpay.comOpenallowedallowedallowedallowedallowedallowedallowedallowedyes
6820odibets.comBlocks some AI searchblockedallowedallowedblockedallowedallowedblockedblocked–
6881scotiabank.comOpenallowedallowedallowedallowedallowedallowedallowedallowed–
6950investors.comBlocks some AI searchpartialpartialpartialblockedblockedblockedblockedblocked–
6953betsson.bet.arUnreadableunknownunknownunknownunknownunknownunknownunknownunknown–
7055sspay01.comUnreadableunknownunknownunknownunknownunknownunknownunknownunknown–
7057betjili365.vipOpenallowedallowedallowedallowedallowedallowedallowedallowed–
7129bankofengland.co.ukUnreadableunknownunknownunknownunknownunknownunknownunknownunknown–
71341jugarbet.comUnreadableunknownunknownunknownunknownunknownunknownunknownunknown–
7196indiafinance.onlineUnreadableunknownunknownunknownunknownunknownunknownunknownunknown–
7205khabar.betOpenallowedallowedallowedallowedallowedallowedallowedallowed–
7207zplaybet.xyzUnreadableunknownunknownunknownunknownunknownunknownunknownunknown–
7233wechatpay.cnNo robots.txtallowedallowedallowedallowedallowedallowedallowedallowed–
7441citibank.comOpenallowedallowedallowedallowedallowedallowedallowedallowed–
7459rabobank.nlUnreadableunknownunknownunknownunknownunknownunknownunknownunknown–
7525nitropay.comOpenallowedallowedallowedallowedallowedallowedallowedallowed–
7594commbank.com.auOpenpartialpartialpartialpartialpartialpartialpartialpartialyes
7608cloudpayments.ruOpenallowedallowedallowedallowedallowedallowedallowedallowed–
7628visitorbet-resmi.comUnreadableunknownunknownunknownunknownunknownunknownunknownunknown–
7687juga-bet-col.comNo robots.txtallowedallowedallowedallowedallowedallowedallowedallowed–
7713betterttv.netNo robots.txtallowedallowedallowedallowedallowedallowedallowedallowed–
7966picpay.comOpenallowedallowedallowedallowedallowedallowedallowedallowedyes
7975profinance.ruOpenallowedallowedallowedallowedallowedallowedallowedallowed–
8033databank.comOpenallowedallowedallowedallowedallowedallowedallowedallowedyes
8104corporatefinanceinstitute.comOpenallowedallowedallowedallowedallowedallowedallowedallowed–
8291skybet.comUnreadableunknownunknownunknownunknownunknownunknownunknownunknown–
8298shopeepay.com.brNo robots.txtallowedallowedallowedallowedallowedallowedallowedallowed–
8359betboom.ruOpenallowedallowedallowedallowedallowedallowedallowedallowed–
8476payu.comNo robots.txtallowedallowedallowedallowedallowedallowedallowedallowed–
8489payscale.comOpenallowedallowedallowedallowedallowedallowedallowedallowed–
8788payoneer.comBlocks some AI searchblockedallowedallowedblockedallowedallowedblockedblockedyes
8873paycor.comOpenallowedallowedallowedallowedallowedallowedallowedallowedyes
9034gobankingrates.comOpenallowedallowedallowedallowedallowedallowedallowedallowed–
9116mrbetlogin.comOpenallowedallowedallowedallowedallowedallowedallowedallowed–
9226otpbank.ruUnreadableunknownunknownunknownunknownunknownunknownunknownunknown–
9364icicibank.comOpenallowedallowedallowedallowedallowedallowedallowedallowed–
9473coinglass.comOpenallowedallowedallowedallowedallowedallowedallowedallowed–
9600kasikornbank.comUnreadableunknownunknownunknownunknownunknownunknownunknownunknown–
9674cryptopro.ruOpenallowedallowedallowedallowedallowedallowedallowedallowed–
9726bitcointalk.orgUnreadableunknownunknownunknownunknownunknownunknownunknownunknown–
9875mrbetgames.comOpenallowedallowedallowedallowedallowedallowedallowedallowed–
9889betonline.agUnreadableunknownunknownunknownunknownunknownunknownunknownunknown–
9959coingraph-update.comNo robots.txtallowedallowedallowedallowedallowedallowedallowedallowed–

FAQ

Do finance websites block AI crawlers?
As of 2026-11-16, 6.9% of the 102 finance websites whose robots.txt we could read block GPTBot, 6.9% block ClaudeBot and 4.9% block PerplexityBot (all sites: 15.4%, 14.7%, 8.2%). 23 of 150 publish an llms.txt file.
Which finance websites block GPTBot?
coindesk.com, paycomonline.net, afterpay.com, bankier.pl, betproexch.live, odibets.com, payoneer.com.

Other categories