AI Crawler Index · category
Do travel websites block AI crawlers?
As of 2026-11-16, 15.4% of the 13 travel websites whose robots.txt we could read block GPTBot, 23.1% block ClaudeBot and 0% block PerplexityBot (all sites: 15.4%, 14.7%, 8.2%). 2 of 14 publish an llms.txt file.
Block rate by crawler: travel vs all sites
| Crawler | Operator | Travel | All sites |
|---|---|---|---|
| ClaudeBot | Anthropic | 23.1% | 14.7% |
| Applebot-Extended | Apple | 23.1% | 13.4% |
| GPTBot | OpenAI | 15.4% | 15.4% |
| Google-CloudVertexBot | 15.4% | 8.8% | |
| MistralAI-Training | Mistral | 15.4% | 7.1% |
| OAI-AdsBot | OpenAI | 7.7% | 3% |
| Claude-SearchBot | Anthropic | 7.7% | 5.7% |
| Claude-User | Anthropic | 7.7% | 5.9% |
| Perplexity-User | Perplexity | 7.7% | 6.1% |
| Google-Extended | 7.7% | 13.2% | |
| Google-GeminiNotebook | 7.7% | 3.3% | |
| Google-Agent | 7.7% | 3.6% | |
| Meta-ExternalAgent | Meta | 7.7% | 13.6% |
| meta-webindexer | Meta | 7.7% | 5.1% |
| meta-externalfetcher | Meta | 7.7% | 6.5% |
| meta-externalads | Meta | 7.7% | 2.9% |
| Amazonbot | Amazon | 7.7% | 13.2% |
| Amzn-SearchBot | Amazon | 7.7% | 4.4% |
| Amzn-User | Amazon | 7.7% | 4.4% |
| DuckAssistBot | DuckDuckGo | 7.7% | 5.9% |
| MistralAI-User | Mistral | 7.7% | 5.7% |
| MistralAI-Index | Mistral | 7.7% | 3.6% |
| CCBot | Common Crawl | 7.7% | 16.4% |
| Bytespider | ByteDance | 7.7% | 16.2% |
| cohere-ai | Cohere | 7.7% | 12.7% |
| OAI-SearchBot | OpenAI | 0% | 5.8% |
| ChatGPT-User | OpenAI | 0% | 7.4% |
| PerplexityBot | Perplexity | 0% | 8.2% |
| Applebot | Apple | 0% | 3.2% |
All 14 travel sites in the index
✕ blocked · ◐ partially blocked (some paths) · ✓ allowed · ? robots.txt unreadable. "#" is the position in our popularity-ordered sample.
| # | Site | Policy | GPTBot | OAI-Search | ChatGPT-User | ClaudeBot | Claude-Search | Perplexity | Google-Ext. | CCBot | llms.txt |
|---|---|---|---|---|---|---|---|---|---|---|---|
| 118 | booking.com | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 261 | tripadvisor.com | Blocks some AI search | blocked | partial | partial | blocked | allowed | partial | blocked | blocked | – |
| 288 | 2gis.com | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 307 | marriott.com | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 326 | uber.com | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 332 | airbnb.com | Blocks training only | blocked | partial | partial | blocked | allowed | partial | allowed | allowed | – |
| 368 | ryanair.com | No robots.txt | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 377 | agoda.com | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 429 | expedia.com | Open | allowed | partial | partial | allowed | partial | partial | allowed | allowed | yes |
| 450 | hilton.com | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 530 | hotpepper.jp | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 577 | amap.com | Blocks some AI search | partial | partial | partial | blocked | blocked | partial | partial | partial | yes |
| 636 | jalan.net | Open | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | – |
| 650 | navitime.co.jp | Unreadable | unknown | unknown | unknown | unknown | unknown | unknown | unknown | unknown | – |
FAQ
- Do travel websites block AI crawlers?
- As of 2026-11-16, 15.4% of the 13 travel websites whose robots.txt we could read block GPTBot, 23.1% block ClaudeBot and 0% block PerplexityBot (all sites: 15.4%, 14.7%, 8.2%). 2 of 14 publish an llms.txt file.
- Which travel websites block GPTBot?
- tripadvisor.com, airbnb.com.
Other categories
- AI GPTBot 10%
- E-commerce GPTBot 14.7%
- Education GPTBot 5.6%
- Entertainment GPTBot 19.6%
- Finance GPTBot 6.9%
- Government GPTBot 3%
- News & media GPTBot 33.7%
- Other GPTBot 15.1%
- Reference GPTBot 27.9%
- Social GPTBot 37.3%
- Tech GPTBot 3.7%
- Telecom GPTBot 0%