TidyToolsAI Crawler Index · data as of · updated daily

AI crawler access · kexart.com

Does kexart.com block GPTBot and other AI crawlers?

GPTBot: blockedAs of 2026-11-16, the robots.txt of kexart.com blocks 28 of 29 AI crawlers on the home page: GPTBot is blocked, ClaudeBot is blocked and PerplexityBot is blocked.

All 29 AI crawlers on kexart.com

Status for the home page (/) under kexart.com's robots.txt. Partially blocked = home page allowed, some paths disallowed.

CrawlerOperatorrobots.txt
Training crawlers
GPTBotOpenAIblocked
ClaudeBotAnthropicblocked
Google-ExtendedGoogleblocked
Applebot-ExtendedAppleblocked
Meta-ExternalAgentMetablocked
AmazonbotAmazonblocked
MistralAI-TrainingMistralblocked
CCBotCommon Crawlblocked
BytespiderByteDanceblocked
cohere-aiCohereblocked
AI search crawlers
OAI-SearchBotOpenAIblocked
Claude-SearchBotAnthropicblocked
PerplexityBotPerplexityblocked
Google-CloudVertexBotGoogleblocked
ApplebotAppleallowed
meta-webindexerMetablocked
Amzn-SearchBotAmazonblocked
DuckAssistBotDuckDuckGoblocked
MistralAI-IndexMistralblocked
User-triggered fetch crawlers
ChatGPT-UserOpenAIblocked
Claude-UserAnthropicblocked
Perplexity-UserPerplexityblocked
Google-GeminiNotebookGoogleblocked
Google-AgentGoogleblocked
meta-externalfetcherMetablocked
Amzn-UserAmazonblocked
MistralAI-UserMistralblocked
Ads review crawlers
OAI-AdsBotOpenAIblocked
meta-externaladsMetablocked

Compared with other other sites in the index: 15.1% of them block GPTBot and 14.1% block ClaudeBot (n=6651). Across all 8,313 readable sites: 15.4% block GPTBot. All other sites →

On .com domains: 15.9% of 4,449 readable sites block GPTBot and 21.2% serve an llms.txt. All .com sites →

History

30 recorded changes since daily tracking started on 2026-10-02:

DateWhatBeforeAfter
2026-11-03OAI-SearchBotallowedblocked
2026-11-03ChatGPT-Userallowedblocked
2026-11-03OAI-AdsBotallowedblocked
2026-11-03Claude-SearchBotallowedblocked
2026-11-03Claude-Userallowedblocked
2026-11-03PerplexityBotallowedblocked
2026-11-03Perplexity-Userallowedblocked
2026-11-03Google-GeminiNotebookallowedblocked
2026-11-03Google-Agentallowedblocked
2026-11-03meta-webindexerallowedblocked
2026-11-03meta-externalfetcherallowedblocked
2026-11-03meta-externaladsallowedblocked
2026-11-03Amzn-SearchBotallowedblocked
2026-11-03Amzn-Userallowedblocked
2026-11-03DuckAssistBotallowedblocked
2026-11-03MistralAI-Userallowedblocked
2026-11-03MistralAI-Indexallowedblocked
2026-11-03(content-signal)search=yes,ai-train=no,use=reference
2026-11-02(content-signal)search=yes,ai-train=no,use=reference
2026-11-01(content-signal)search=yes,ai-train=no,use=reference
2026-10-31(content-signal)search=yes,ai-train=no,use=reference
2026-10-30(content-signal)search=yes,ai-train=no,use=reference
2026-10-29(content-signal)search=yes,ai-train=no,use=reference
2026-10-28(content-signal)search=yes,ai-train=no,use=reference
2026-10-27(content-signal)search=yes,ai-train=no,use=reference
2026-10-26(content-signal)search=yes,ai-train=no,use=reference
2026-10-25(content-signal)search=yes,ai-train=no,use=reference
2026-10-24(content-signal)search=yes,ai-train=no,use=reference
2026-10-23(content-signal)search=yes,ai-train=no,use=reference
2026-10-22(content-signal)search=yes,ai-train=no,use=reference

FAQ

Does kexart.com block GPTBot?
Yes. As of 2026-11-16, https://kexart.com/robots.txt disallows the home page for GPTBot, OpenAI's AI training crawler.
Does kexart.com block ClaudeBot?
Yes. As of 2026-11-16, https://kexart.com/robots.txt disallows the home page for ClaudeBot, Anthropic's AI training crawler.
Does kexart.com block PerplexityBot?
Yes. As of 2026-11-16, https://kexart.com/robots.txt disallows the home page for PerplexityBot, Perplexity's AI search crawler.
Can ChatGPT search (OAI-SearchBot) crawl kexart.com?
Yes. As of 2026-11-16, https://kexart.com/robots.txt disallows the home page for OAI-SearchBot, OpenAI's AI search crawler.
Does kexart.com block Google-Extended (Gemini training)?
Yes. As of 2026-11-16, https://kexart.com/robots.txt disallows the home page for Google-Extended, Google's AI training crawler.
Does kexart.com have an llms.txt file?
No. As of 2026-11-16, kexart.com does not serve a plain-text /llms.txt file.
How can I check kexart.com or my own site again?
Use the free AI crawler checker at tools.yukai.uk/free/ai-crawler-checker (no sign-up). To check many sites and get alerts when rules change, use the AI Crawler Access Checker on Apify.

Similar .com sites

Which .com sites block GPTBot · What is kexart.com built with? (Tech Stack Index)

All 10,000 sites · All 29 AI crawlers · AI Crawler Index