TidyToolsAI Crawler Index · data as of · updated daily

AI crawler access · samehadaku.how

Does samehadaku.how block GPTBot and other AI crawlers?

GPTBot: blockedAs of 2026-11-16, the robots.txt of samehadaku.how blocks 11 of 29 AI crawlers on the home page: GPTBot is blocked, ClaudeBot is blocked and PerplexityBot is allowed.

All 29 AI crawlers on samehadaku.how

Status for the home page (/) under samehadaku.how's robots.txt. Partially blocked = home page allowed, some paths disallowed.

CrawlerOperatorrobots.txt
Training crawlers
GPTBotOpenAIblocked
ClaudeBotAnthropicblocked
Google-ExtendedGoogleblocked
Applebot-ExtendedAppleblocked
Meta-ExternalAgentMetablocked
AmazonbotAmazonblocked
MistralAI-TrainingMistralblocked
CCBotCommon Crawlblocked
BytespiderByteDanceblocked
cohere-aiCohereblocked
AI search crawlers
OAI-SearchBotOpenAIallowed
Claude-SearchBotAnthropicallowed
PerplexityBotPerplexityallowed
Google-CloudVertexBotGoogleblocked
ApplebotAppleallowed
meta-webindexerMetaallowed
Amzn-SearchBotAmazonallowed
DuckAssistBotDuckDuckGoallowed
MistralAI-IndexMistralallowed
User-triggered fetch crawlers
ChatGPT-UserOpenAIallowed
Claude-UserAnthropicallowed
Perplexity-UserPerplexityallowed
Google-GeminiNotebookGoogleallowed
Google-AgentGoogleallowed
meta-externalfetcherMetaallowed
Amzn-UserAmazonallowed
MistralAI-UserMistralallowed
Ads review crawlers
OAI-AdsBotOpenAIallowed
meta-externaladsMetaallowed

Compared with other other sites in the index: 15.1% of them block GPTBot and 14.1% block ClaudeBot (n=6651). Across all 8,313 readable sites: 15.4% block GPTBot. All other sites →

History

30 recorded changes since daily tracking started on 2026-10-02:

DateWhatBeforeAfter
2026-10-24GPTBotallowedblocked
2026-10-24ClaudeBotallowedblocked
2026-10-24Google-Extendedallowedblocked
2026-10-24Google-CloudVertexBotallowedblocked
2026-10-24Applebot-Extendedallowedblocked
2026-10-24Meta-ExternalAgentallowedblocked
2026-10-24Amazonbotallowedblocked
2026-10-24MistralAI-Trainingallowedblocked
2026-10-24CCBotallowedblocked
2026-10-24Bytespiderallowedblocked
2026-10-24cohere-aiallowedblocked
2026-10-24overall policyOpenBlocks some AI search
2026-10-24(content-signal)search=yes,ai-train=no,use=reference
2026-10-05GPTBotblockedallowed
2026-10-05OAI-SearchBotblockedallowed
2026-10-05ChatGPT-Userblockedallowed
2026-10-05OAI-AdsBotblockedallowed
2026-10-05ClaudeBotblockedallowed
2026-10-05Claude-SearchBotblockedallowed
2026-10-05Claude-Userblockedallowed
2026-10-05PerplexityBotblockedallowed
2026-10-05Perplexity-Userblockedallowed
2026-10-05Google-Extendedblockedallowed
2026-10-05Google-CloudVertexBotblockedallowed
2026-10-05Google-GeminiNotebookblockedallowed
2026-10-05Google-Agentblockedallowed
2026-10-05Applebotblockedallowed
2026-10-05Applebot-Extendedblockedallowed
2026-10-05Meta-ExternalAgentblockedallowed
2026-10-05meta-webindexerblockedallowed

FAQ

Does samehadaku.how block GPTBot?
Yes. As of 2026-11-16, https://samehadaku.how/robots.txt disallows the home page for GPTBot, OpenAI's AI training crawler.
Does samehadaku.how block ClaudeBot?
Yes. As of 2026-11-16, https://samehadaku.how/robots.txt disallows the home page for ClaudeBot, Anthropic's AI training crawler.
Does samehadaku.how block PerplexityBot?
No. As of 2026-11-16, https://samehadaku.how/robots.txt allows PerplexityBot, Perplexity's AI search crawler, on the home page.
Can ChatGPT search (OAI-SearchBot) crawl samehadaku.how?
No. As of 2026-11-16, https://samehadaku.how/robots.txt allows OAI-SearchBot, OpenAI's AI search crawler, on the home page.
Does samehadaku.how block Google-Extended (Gemini training)?
Yes. As of 2026-11-16, https://samehadaku.how/robots.txt disallows the home page for Google-Extended, Google's AI training crawler.
Does samehadaku.how have an llms.txt file?
No. As of 2026-11-16, samehadaku.how does not serve a plain-text /llms.txt file.
How can I check samehadaku.how or my own site again?
Use the free AI crawler checker at tools.yukai.uk/free/ai-crawler-checker (no sign-up). To check many sites and get alerts when rules change, use the AI Crawler Access Checker on Apify.

What is samehadaku.how built with? (Tech Stack Index)

All 10,000 sites · All 29 AI crawlers · AI Crawler Index