TidyToolsAI Crawler Index · data as of · updated daily

AI crawler access · razest.net

Does razest.net block GPTBot and other AI crawlers?

We could not read the robots.txt of razest.net from our datacenter IP on 2026-11-16 (an error page, a bot wall or a connection problem), so its AI crawler rules are unknown.

All 29 AI crawlers on razest.net

Status for the home page (/) under razest.net's robots.txt. Partially blocked = home page allowed, some paths disallowed.

CrawlerOperatorrobots.txt
Training crawlers
GPTBotOpenAIunknown
ClaudeBotAnthropicunknown
Google-ExtendedGoogleunknown
Applebot-ExtendedAppleunknown
Meta-ExternalAgentMetaunknown
AmazonbotAmazonunknown
MistralAI-TrainingMistralunknown
CCBotCommon Crawlunknown
BytespiderByteDanceunknown
cohere-aiCohereunknown
AI search crawlers
OAI-SearchBotOpenAIunknown
Claude-SearchBotAnthropicunknown
PerplexityBotPerplexityunknown
Google-CloudVertexBotGoogleunknown
ApplebotAppleunknown
meta-webindexerMetaunknown
Amzn-SearchBotAmazonunknown
DuckAssistBotDuckDuckGounknown
MistralAI-IndexMistralunknown
User-triggered fetch crawlers
ChatGPT-UserOpenAIunknown
Claude-UserAnthropicunknown
Perplexity-UserPerplexityunknown
Google-GeminiNotebookGoogleunknown
Google-AgentGoogleunknown
meta-externalfetcherMetaunknown
Amzn-UserAmazonunknown
MistralAI-UserMistralunknown
Ads review crawlers
OAI-AdsBotOpenAIunknown
meta-externaladsMetaunknown

History

30 recorded changes since daily tracking started on 2026-10-02:

DateWhatBeforeAfter
2026-10-30GPTBotallowedblocked
2026-10-30OAI-SearchBotallowedblocked
2026-10-30ChatGPT-Userallowedblocked
2026-10-30OAI-AdsBotallowedblocked
2026-10-30ClaudeBotallowedblocked
2026-10-30Claude-SearchBotallowedblocked
2026-10-30Claude-Userallowedblocked
2026-10-30PerplexityBotallowedblocked
2026-10-30Perplexity-Userallowedblocked
2026-10-30Google-Extendedallowedblocked
2026-10-30Google-CloudVertexBotallowedblocked
2026-10-30Google-GeminiNotebookallowedblocked
2026-10-30Google-Agentallowedblocked
2026-10-30Applebotallowedblocked
2026-10-30Applebot-Extendedallowedblocked
2026-10-30Meta-ExternalAgentallowedblocked
2026-10-30meta-webindexerallowedblocked
2026-10-30meta-externalfetcherallowedblocked
2026-10-30meta-externaladsallowedblocked
2026-10-30Amazonbotallowedblocked
2026-10-30Amzn-SearchBotallowedblocked
2026-10-30Amzn-Userallowedblocked
2026-10-30DuckAssistBotallowedblocked
2026-10-30MistralAI-Userallowedblocked
2026-10-30MistralAI-Indexallowedblocked
2026-10-30MistralAI-Trainingallowedblocked
2026-10-30CCBotallowedblocked
2026-10-30Bytespiderallowedblocked
2026-10-30cohere-aiallowedblocked
2026-10-30overall policyNo robots.txtBlocks all AI search

FAQ

Does razest.net block GPTBot?
Unknown. https://razest.net/robots.txt could not be read on 2026-11-16.
Does razest.net block ClaudeBot?
Unknown. https://razest.net/robots.txt could not be read on 2026-11-16.
Does razest.net block PerplexityBot?
Unknown. https://razest.net/robots.txt could not be read on 2026-11-16.
Can ChatGPT search (OAI-SearchBot) crawl razest.net?
Unknown. https://razest.net/robots.txt could not be read on 2026-11-16.
Does razest.net block Google-Extended (Gemini training)?
Unknown. https://razest.net/robots.txt could not be read on 2026-11-16.
How can I check razest.net or my own site again?
Use the free AI crawler checker at tools.yukai.uk/free/ai-crawler-checker (no sign-up). To check many sites and get alerts when rules change, use the AI Crawler Access Checker on Apify.

Similar .net sites

All 10,000 sites · All 29 AI crawlers · AI Crawler Index