TidyToolsAI Crawler Index · data as of · updated daily

AI Crawler Index · country

Do websites in Australia (.au) block AI crawlers?

As of 2026-10-03, 27% of the 37 websites in Australia (.au) whose robots.txt we could read block GPTBot, 29.7% block ClaudeBot and 27% block PerplexityBot (all 8,357 sites: 12%, 11%, 8.3%). 6 of them (16.2%) serve an llms.txt (all sites: 15.2%).

27%
block GPTBot (all sites 12%)
29.7%
block ClaudeBot (all sites 11%)
16.2%
serve an llms.txt (all sites 15.2%)
37
of 56 sites with a readable robots.txt

Block rate by crawler: .au (Australia) vs all sites

CrawlerOperator.au (Australia)All sitesDifference
BytespiderByteDance29.7%12.3%+17.4 pp
CCBotCommon Crawl29.7%12.8%+16.9 pp
ClaudeBotAnthropic29.7%11%+18.7 pp
cohere-aiCohere29.7%9.1%+20.6 pp
Meta-ExternalAgentMeta29.7%9.5%+20.2 pp
AmazonbotAmazon27%9.2%+17.8 pp
Applebot-ExtendedApple27%9.6%+17.4 pp
Google-ExtendedGoogle27%9.5%+17.5 pp
GPTBotOpenAI27%12%+15 pp
PerplexityBotPerplexity27%8.3%+18.7 pp
meta-externalfetcherMeta24.3%6%+18.3 pp
ChatGPT-UserOpenAI21.6%7.5%+14.1 pp
DuckAssistBotDuckDuckGo21.6%5.8%+15.8 pp
OAI-SearchBotOpenAI18.9%5.7%+13.2 pp
Claude-SearchBotAnthropic16.2%5.4%+10.8 pp
Claude-UserAnthropic16.2%5.5%+10.7 pp
Google-GeminiNotebookGoogle16.2%3%+13.2 pp
Perplexity-UserPerplexity16.2%5.7%+10.5 pp
Amzn-SearchBotAmazon13.5%4.1%+9.4 pp
Amzn-UserAmazon13.5%3.9%+9.6 pp
Google-AgentGoogle13.5%3.2%+10.3 pp
Google-CloudVertexBotGoogle13.5%5.2%+8.3 pp
meta-externaladsMeta13.5%2.8%+10.7 pp
meta-webindexerMeta13.5%4.6%+8.9 pp
MistralAI-IndexMistral13.5%3.3%+10.2 pp
MistralAI-TrainingMistral13.5%3.3%+10.2 pp
MistralAI-UserMistral13.5%5.4%+8.1 pp
OAI-AdsBotOpenAI13.5%3%+10.5 pp
ApplebotApple2.7%3.1%-0.4 pp

Who blocks what, by name: GPTBot, ClaudeBot, PerplexityBot, Google-Extended, CCBot, Bytespider, Applebot-Extended, OAI-SearchBot.

Policy mix

PolicySitesShare
Open2646.4%
Unreadable1933.9%
Blocks some AI search1017.9%
Blocks all AI search11.8%

Most restrictive .au (Australia) sites

  1. swin.edu.au 29 of 29 blocked
  2. news.com.au 27 of 29 blocked
  3. dailytelegraph.com.au 27 of 29 blocked
  4. heraldsun.com.au 27 of 29 blocked
  5. theaustralian.com.au 24 of 29 blocked
  6. smh.com.au 15 of 29 blocked
  7. nine.com.au 15 of 29 blocked
  8. theage.com.au 15 of 29 blocked
  9. realestate.com.au 12 of 29 blocked
  10. sbs.com.au 12 of 29 blocked
  11. abc.net.au 9 of 29 blocked

Sites with an llms.txt (6)

All 56 .au (Australia) sites

✕ blocked · ◐ partially blocked (some paths) · ✓ allowed · ? robots.txt unreadable. "#" is the position in our popularity-ordered sample.

#SitePolicyGPTBotOAI-SearchChatGPT-UserClaudeBotClaude-SearchPerplexityGoogle-Ext.CCBotllms.txt
595abc.net.auBlocks some AI searchblockedallowedblockedblockedallowedblockedblockedblocked–
1327news.com.auBlocks some AI searchblockedblockedblockedblockedblockedblockedblockedblocked–
1351smh.com.auBlocks some AI searchblockedblockedblockedblockedallowedblockedblockedblocked–
1415nsw.gov.auOpenallowedallowedallowedallowedallowedallowedallowedallowedyes
1517ventraip.net.auOpenallowedallowedallowedallowedallowedallowedallowedallowed–
1605unimelb.edu.auUnreadableunknownunknownunknownunknownunknownunknownunknownunknown–
1718aarnet.net.auUnreadableunknownunknownunknownunknownunknownunknownunknownunknown–
1797iinet.net.auOpenallowedallowedallowedallowedallowedallowedallowedallowed–
2056realestate.com.auBlocks some AI searchblockedallowedallowedblockedblockedblockedallowedblocked–
2079anu.edu.auOpenallowedallowedallowedallowedallowedallowedallowedallowed–
2281unsw.edu.auUnreadableunknownunknownunknownunknownunknownunknownunknownunknown–
2320theiconic.com.auUnreadableunknownunknownunknownunknownunknownunknownunknownunknown–
2618uq.edu.auOpenallowedallowedallowedallowedallowedallowedallowedallowed–
2637nine.com.auBlocks some AI searchblockedblockedblockedblockedallowedblockedblockedblocked–
2804sydney.edu.auOpenallowedallowedallowedallowedallowedallowedallowedallowed–
2939theage.com.auBlocks some AI searchblockedblockedblockedblockedallowedblockedblockedblocked–
3137nla.gov.auOpenallowedallowedallowedallowedallowedallowedallowedallowed–
3561sbs.com.auBlocks some AI searchblockedpartialpartialblockedpartialpartialblockedblocked–
3729csiro.auUnreadableunknownunknownunknownunknownunknownunknownunknownunknown–
3773bunnings.com.auUnreadableunknownunknownunknownunknownunknownunknownunknownunknown–
3892adelaide.edu.auOpenallowedallowedallowedallowedallowedallowedallowedallowed–
4496theaustralian.com.auBlocks some AI searchallowedallowedallowedblockedblockedblockedblockedblocked–
4971bom.gov.auUnreadableunknownunknownunknownunknownunknownunknownunknownunknown–
5171uwa.edu.auOpenallowedallowedallowedallowedallowedallowedallowedallowed–
5231newcastle.edu.auUnreadableunknownunknownunknownunknownunknownunknownunknownunknown–
5781uts.edu.auOpenallowedallowedallowedallowedallowedallowedallowedallowed–
6330deakin.edu.auUnreadableunknownunknownunknownunknownunknownunknownunknownunknown–
6886mq.edu.auUnreadableunknownunknownunknownunknownunknownunknownunknownunknown–
6973coles.com.auUnreadableunknownunknownunknownunknownunknownunknownunknownunknown–
7198kmart.com.auUnreadableunknownunknownunknownunknownunknownunknownunknownunknown–
7295bellybelly.com.auOpenallowedallowedallowedallowedallowedallowedallowedallowed–
7434auspost.com.auOpenallowedallowedallowedallowedallowedallowedallowedallowed–
7516latrobe.edu.auOpenallowedallowedallowedallowedallowedallowedallowedallowedyes
7540nab.com.auOpenallowedallowedallowedallowedallowedallowedallowedallowedyes
7594commbank.com.auOpenpartialpartialpartialpartialpartialpartialpartialpartialyes
8007rmit.edu.auOpenallowedallowedallowedallowedallowedallowedallowedallowed–
8095homeaffairs.gov.auUnreadableunknownunknownunknownunknownunknownunknownunknownunknown–
8105uow.edu.auOpenallowedallowedallowedallowedallowedallowedallowedallowed–
8170usyd.edu.auOpenallowedallowedallowedallowedallowedallowedallowedallowed–
8203qut.edu.auUnreadableunknownunknownunknownunknownunknownunknownunknownunknown–
8411austlii.edu.auUnreadableunknownunknownunknownunknownunknownunknownunknownunknown–
8460ato.gov.auUnreadableunknownunknownunknownunknownunknownunknownunknownunknown–
8463my.gov.auOpenallowedallowedallowedallowedallowedallowedallowedallowed–
8530abs.gov.auOpenallowedallowedallowedallowedallowedallowedallowedallowed–
8547jbhifi.com.auOpenallowedallowedallowedallowedallowedallowedallowedallowedyes
8674oaic.gov.auOpenallowedallowedallowedallowedallowedallowedallowedallowed–
8682woolworths.com.auUnreadableunknownunknownunknownunknownunknownunknownunknownunknown–
8739westpac.com.auOpenallowedallowedallowedallowedallowedallowedallowedallowedyes
8741iprimus.net.auUnreadableunknownunknownunknownunknownunknownunknownunknownunknown–
9096dailytelegraph.com.auBlocks some AI searchblockedblockedblockedblockedblockedblockedblockedblocked–
9332crucial.com.auOpenallowedallowedallowedallowedallowedallowedallowedallowed–
9556healthdirect.gov.auOpenallowedallowedallowedallowedallowedallowedallowedallowed–
9562afl.com.auOpenallowedallowedallowedallowedallowedallowedallowedallowed–
9622aph.gov.auUnreadableunknownunknownunknownunknownunknownunknownunknownunknown–
9683heraldsun.com.auBlocks some AI searchblockedblockedblockedblockedblockedblockedblockedblocked–
9907swin.edu.auBlocks all AI searchblockedblockedblockedblockedblockedblockedblockedblocked–

Categories on .au (Australia)

FAQ

Do websites in Australia (.au) block AI crawlers?
As of 2026-10-03, 27% of the 37 websites in Australia (.au) whose robots.txt we could read block GPTBot, 29.7% block ClaudeBot and 27% block PerplexityBot (all 8,357 sites: 12%, 11%, 8.3%). 6 of them (16.2%) serve an llms.txt (all sites: 15.2%).
Which .au (Australia) websites block the most AI crawlers?
swin.edu.au (29 of 29), news.com.au (27 of 29), dailytelegraph.com.au (27 of 29), heraldsun.com.au (27 of 29), theaustralian.com.au (24 of 29). The full per-site status is on each site page.
How is this measured?
We fetch each site's robots.txt every day from one residential IP in Taiwan with an honest user agent, and apply the rules to 29 AI crawlers for the home page. 56 .au (Australia) sites are in the sample, 37 had a readable robots.txt (19 did not answer or returned an error page).

Other countries and suffixes

Compare all suffixes