TidyToolsAI Crawler Index · data as of · updated daily

AI Crawler Index · category

Do travel websites block AI crawlers?

As of 2026-11-16, 15.4% of the 13 travel websites whose robots.txt we could read block GPTBot, 23.1% block ClaudeBot and 0% block PerplexityBot (all sites: 15.4%, 14.7%, 8.2%). 2 of 14 publish an llms.txt file.

Block rate by crawler: travel vs all sites

CrawlerOperatorTravelAll sites
ClaudeBotAnthropic23.1%14.7%
Applebot-ExtendedApple23.1%13.4%
GPTBotOpenAI15.4%15.4%
Google-CloudVertexBotGoogle15.4%8.8%
MistralAI-TrainingMistral15.4%7.1%
OAI-AdsBotOpenAI7.7%3%
Claude-SearchBotAnthropic7.7%5.7%
Claude-UserAnthropic7.7%5.9%
Perplexity-UserPerplexity7.7%6.1%
Google-ExtendedGoogle7.7%13.2%
Google-GeminiNotebookGoogle7.7%3.3%
Google-AgentGoogle7.7%3.6%
Meta-ExternalAgentMeta7.7%13.6%
meta-webindexerMeta7.7%5.1%
meta-externalfetcherMeta7.7%6.5%
meta-externaladsMeta7.7%2.9%
AmazonbotAmazon7.7%13.2%
Amzn-SearchBotAmazon7.7%4.4%
Amzn-UserAmazon7.7%4.4%
DuckAssistBotDuckDuckGo7.7%5.9%
MistralAI-UserMistral7.7%5.7%
MistralAI-IndexMistral7.7%3.6%
CCBotCommon Crawl7.7%16.4%
BytespiderByteDance7.7%16.2%
cohere-aiCohere7.7%12.7%
OAI-SearchBotOpenAI0%5.8%
ChatGPT-UserOpenAI0%7.4%
PerplexityBotPerplexity0%8.2%
ApplebotApple0%3.2%

All 14 travel sites in the index

✕ blocked · ◐ partially blocked (some paths) · ✓ allowed · ? robots.txt unreadable. "#" is the position in our popularity-ordered sample.

#SitePolicyGPTBotOAI-SearchChatGPT-UserClaudeBotClaude-SearchPerplexityGoogle-Ext.CCBotllms.txt
118booking.comOpenallowedallowedallowedallowedallowedallowedallowedallowed–
261tripadvisor.comBlocks some AI searchblockedpartialpartialblockedallowedpartialblockedblocked–
2882gis.comOpenallowedallowedallowedallowedallowedallowedallowedallowed–
307marriott.comOpenallowedallowedallowedallowedallowedallowedallowedallowed–
326uber.comOpenallowedallowedallowedallowedallowedallowedallowedallowed–
332airbnb.comBlocks training onlyblockedpartialpartialblockedallowedpartialallowedallowed–
368ryanair.comNo robots.txtallowedallowedallowedallowedallowedallowedallowedallowed–
377agoda.comOpenallowedallowedallowedallowedallowedallowedallowedallowed–
429expedia.comOpenallowedpartialpartialallowedpartialpartialallowedallowedyes
450hilton.comOpenallowedallowedallowedallowedallowedallowedallowedallowed–
530hotpepper.jpOpenallowedallowedallowedallowedallowedallowedallowedallowed–
577amap.comBlocks some AI searchpartialpartialpartialblockedblockedpartialpartialpartialyes
636jalan.netOpenallowedallowedallowedallowedallowedallowedallowedallowed–
650navitime.co.jpUnreadableunknownunknownunknownunknownunknownunknownunknownunknown–

FAQ

Do travel websites block AI crawlers?
As of 2026-11-16, 15.4% of the 13 travel websites whose robots.txt we could read block GPTBot, 23.1% block ClaudeBot and 0% block PerplexityBot (all sites: 15.4%, 14.7%, 8.2%). 2 of 14 publish an llms.txt file.
Which travel websites block GPTBot?
tripadvisor.com, airbnb.com.

Other categories