TidyToolsAI Crawler Index · data as of · updated daily

AI crawler access · mbl.is

Does mbl.is block GPTBot and other AI crawlers?

We could not read the robots.txt of mbl.is from our datacenter IP on 2026-11-16 (an error page, a bot wall or a connection problem), so its AI crawler rules are unknown.

All 29 AI crawlers on mbl.is

Status for the home page (/) under mbl.is's robots.txt. Partially blocked = home page allowed, some paths disallowed.

CrawlerOperatorrobots.txt
Training crawlers
GPTBotOpenAIunknown
ClaudeBotAnthropicunknown
Google-ExtendedGoogleunknown
Applebot-ExtendedAppleunknown
Meta-ExternalAgentMetaunknown
AmazonbotAmazonunknown
MistralAI-TrainingMistralunknown
CCBotCommon Crawlunknown
BytespiderByteDanceunknown
cohere-aiCohereunknown
AI search crawlers
OAI-SearchBotOpenAIunknown
Claude-SearchBotAnthropicunknown
PerplexityBotPerplexityunknown
Google-CloudVertexBotGoogleunknown
ApplebotAppleunknown
meta-webindexerMetaunknown
Amzn-SearchBotAmazonunknown
DuckAssistBotDuckDuckGounknown
MistralAI-IndexMistralunknown
User-triggered fetch crawlers
ChatGPT-UserOpenAIunknown
Claude-UserAnthropicunknown
Perplexity-UserPerplexityunknown
Google-GeminiNotebookGoogleunknown
Google-AgentGoogleunknown
meta-externalfetcherMetaunknown
Amzn-UserAmazonunknown
MistralAI-UserMistralunknown
Ads review crawlers
OAI-AdsBotOpenAIunknown
meta-externaladsMetaunknown

History

30 recorded changes since daily tracking started on 2026-10-02:

DateWhatBeforeAfter
2026-10-05GPTBotblockedallowed
2026-10-05OAI-SearchBotblockedallowed
2026-10-05ChatGPT-Userblockedallowed
2026-10-05OAI-AdsBotblockedallowed
2026-10-05ClaudeBotblockedallowed
2026-10-05Claude-SearchBotblockedallowed
2026-10-05Claude-Userblockedallowed
2026-10-05PerplexityBotblockedallowed
2026-10-05Perplexity-Userblockedallowed
2026-10-05Google-Extendedblockedallowed
2026-10-05Google-CloudVertexBotblockedallowed
2026-10-05Google-GeminiNotebookblockedallowed
2026-10-05Google-Agentblockedallowed
2026-10-05Applebotblockedallowed
2026-10-05Applebot-Extendedblockedallowed
2026-10-05Meta-ExternalAgentblockedallowed
2026-10-05meta-webindexerblockedallowed
2026-10-05meta-externalfetcherblockedallowed
2026-10-05meta-externaladsblockedallowed
2026-10-05Amazonbotblockedallowed
2026-10-05Amzn-SearchBotblockedallowed
2026-10-05Amzn-Userblockedallowed
2026-10-05DuckAssistBotblockedallowed
2026-10-05MistralAI-Userblockedallowed
2026-10-05MistralAI-Indexblockedallowed
2026-10-05MistralAI-Trainingblockedallowed
2026-10-05CCBotblockedallowed
2026-10-05Bytespiderblockedallowed
2026-10-05cohere-aiblockedallowed
2026-10-05overall policyBlocks all AI searchOpen

FAQ

Does mbl.is block GPTBot?
Unknown. https://mbl.is/robots.txt could not be read on 2026-11-16.
Does mbl.is block ClaudeBot?
Unknown. https://mbl.is/robots.txt could not be read on 2026-11-16.
Does mbl.is block PerplexityBot?
Unknown. https://mbl.is/robots.txt could not be read on 2026-11-16.
Can ChatGPT search (OAI-SearchBot) crawl mbl.is?
Unknown. https://mbl.is/robots.txt could not be read on 2026-11-16.
Does mbl.is block Google-Extended (Gemini training)?
Unknown. https://mbl.is/robots.txt could not be read on 2026-11-16.
How can I check mbl.is or my own site again?
Use the free AI crawler checker at tools.yukai.uk/free/ai-crawler-checker (no sign-up). To check many sites and get alerts when rules change, use the AI Crawler Access Checker on Apify.

What is mbl.is built with? (Tech Stack Index)

All 10,000 sites · All 29 AI crawlers · AI Crawler Index