TidyToolsAI Crawler Index · data as of · updated daily

AI crawler access · theaustralian.com.au

Does theaustralian.com.au block GPTBot and other AI crawlers?

GPTBot: allowedAs of 2026-10-03, the robots.txt of theaustralian.com.au blocks 24 of 29 AI crawlers on the home page: GPTBot is allowed, ClaudeBot is blocked and PerplexityBot is blocked.

All 29 AI crawlers on theaustralian.com.au

Status for the home page (/) under theaustralian.com.au's robots.txt. Partially blocked = home page allowed, some paths disallowed.

CrawlerOperatorrobots.txt
Training crawlers
GPTBotOpenAIallowed
ClaudeBotAnthropicblocked
Google-ExtendedGoogleblocked
Applebot-ExtendedAppleblocked
Meta-ExternalAgentMetablocked
AmazonbotAmazonblocked
MistralAI-TrainingMistralblocked
CCBotCommon Crawlblocked
BytespiderByteDanceblocked
cohere-aiCohereblocked
AI search crawlers
OAI-SearchBotOpenAIallowed
Claude-SearchBotAnthropicblocked
PerplexityBotPerplexityblocked
Google-CloudVertexBotGoogleallowed
ApplebotAppleallowed
meta-webindexerMetablocked
Amzn-SearchBotAmazonblocked
DuckAssistBotDuckDuckGoblocked
MistralAI-IndexMistralblocked
User-triggered fetch crawlers
ChatGPT-UserOpenAIallowed
Claude-UserAnthropicblocked
Perplexity-UserPerplexityblocked
Google-GeminiNotebookGoogleblocked
Google-AgentGoogleblocked
meta-externalfetcherMetablocked
Amzn-UserAmazonblocked
MistralAI-UserMistralblocked
Ads review crawlers
OAI-AdsBotOpenAIblocked
meta-externaladsMetablocked

Compared with other other sites in the index: 11.6% of them block GPTBot and 10.4% block ClaudeBot (n=6685). Across all 8,357 readable sites: 12% block GPTBot. All other sites →

On .au (Australia) domains: 27% of 37 readable sites block GPTBot and 16.2% serve an llms.txt. All .au (Australia) sites →

History

No change recorded since daily tracking started on 2026-10-02. We re-check theaustralian.com.au every day; a change in its AI crawler rules shows up here and on the index page.

FAQ

Does theaustralian.com.au block GPTBot?
No. As of 2026-10-03, https://theaustralian.com.au/robots.txt allows GPTBot, OpenAI's AI training crawler, on the home page.
Does theaustralian.com.au block ClaudeBot?
Yes. As of 2026-10-03, https://theaustralian.com.au/robots.txt disallows the home page for ClaudeBot, Anthropic's AI training crawler.
Does theaustralian.com.au block PerplexityBot?
Yes. As of 2026-10-03, https://theaustralian.com.au/robots.txt disallows the home page for PerplexityBot, Perplexity's AI search crawler.
Can ChatGPT search (OAI-SearchBot) crawl theaustralian.com.au?
No. As of 2026-10-03, https://theaustralian.com.au/robots.txt allows OAI-SearchBot, OpenAI's AI search crawler, on the home page.
Does theaustralian.com.au block Google-Extended (Gemini training)?
Yes. As of 2026-10-03, https://theaustralian.com.au/robots.txt disallows the home page for Google-Extended, Google's AI training crawler.
Does theaustralian.com.au have an llms.txt file?
No. As of 2026-10-03, theaustralian.com.au does not serve a plain-text /llms.txt file.
How can I check theaustralian.com.au or my own site again?
Use the free AI crawler checker at tools.yukai.uk/free/ai-crawler-checker (no sign-up). To check many sites and get alerts when rules change, use the AI Crawler Access Checker on Apify.

Similar .au (Australia) sites

Which .au (Australia) sites block GPTBot

All 10,000 sites · All 29 AI crawlers · AI Crawler Index