Qwairy
Metrics & Analytics

Crawler Occurrences

Number of observed HTTP requests from documented AI crawler user agents in a site's connected server logs.

What is Crawler Occurrences?

Crawler Occurrences measure requests attributed to documented AI user agents such as GPTBot, ClaudeBot, or PerplexityBot in the logs a site delivers. They show which page was requested, when, by which classified crawler, and which HTTP response group was returned. A high count is not evidence that a platform values, indexed, trained on, retrieved, or cited that page. Missing delivery, provider filters, quotas, and bounded page detail can reduce the visible count, so coverage metadata must be read with the metric.

How Qwairy Makes This Actionable

Qwairy classifies observable AI user agents in connected server logs and consolidates them into retained daily and monthly snapshots. See exact observed totals, frequency, page groups, HTTP responses, and explicit coverage caveats without retaining raw traffic events.

Frequently Asked Questions

No universal threshold exists, and crawler volume alone is not a visibility score. Compare like-for-like periods only when log coverage is complete, then use citation monitoring and AI referral analytics as separate outcome signals. A change in crawl volume can reflect platform scheduling, site structure, provider delivery, or content changes.

Different operators and user agents have different documented purposes and crawl schedules. A lower count may reflect robots.txt rules, HTTP errors, platform behavior, or incomplete log delivery. Check the crawler's documentation, your access rules, response groups, and coverage status before drawing a conclusion.

You can remove access barriers by reviewing robots.txt, authentication, rate limits, redirects, and error responses, and by keeping pages discoverable through normal site architecture. More crawler requests do not automatically improve visibility, so validate outcomes separately through citation monitoring and AI referral traffic.
Share:

Related Terms

AI Crawler

Technical

Automated web client operated by an AI company for model training, AI search, or user-requested retrieval.

GPTBot

Technical

OpenAI's web crawler for content that may be used to train generative AI models, separate from the OAI-SearchBot search crawler.

ClaudeBot

Technical

Anthropic's web crawler that collects public web content to train and improve Claude models; site owners control its access via robots.txt.

Pages Performance

Metrics & Analytics

Page-level view combining observed crawler requests, AI referral sessions, search data, and citation signals when those sources are connected.

Previous
Crawlability

How easily search engine and AI crawlers like GPTBot or ClaudeBot can access, navigate, and index a website's content.

Next
Credit

Consumption unit used to execute operations in a GEO platform (answer generation, analyses).

← Back to GlossaryLast updated Jun 9, 2026