Anthropic's web crawler that collects public web content to train and improve Claude models; site owners control its access via robots.txt.
ClaudeBot is Anthropic's web crawler, identified by the user-agent 'ClaudeBot'. It collects data to train and improve Claude models. As with GPTBot, you can control its access via robots.txt. Allowing ClaudeBot enables your content to be considered in the Claude ecosystem.
Qwairy classifies observed ClaudeBot requests in connected server logs. Monitor retained request frequency, requested page groups, HTTP responses, and coverage without treating a request as proof of indexing or priority.
Automated web client operated by an AI company for model training, AI search, or user-requested retrieval.
Language model developed by Anthropic, designed to be helpful, harmless, and honest.
Text file placed at the root of a website to indicate to indexing robots which pages to explore or avoid.