Connect AI Crawler Visits, Citations, and Referral Traffic

Every page a bot fetched and no engine has cited raises one question: whose problem is it, engineering's, editorial's, or nobody's yet. This workflow keeps four data layers apart, then answers that question per page with a rule that stops at the first match.

The GEO problem it solves

The compression happens twice: crawled therefore processed, cited therefore clicked. get_crawler_activity is built from a bounded bot state rather than retained HTTP events, and says so: each bot with a first-seen and last-seen date, plus an explicit note that raw events are unavailable. No status codes, no response bodies, no per-page counts, no period input. And an AI-referred session never names the citation behind the click.

The windows differ mechanically too, the quiet source of bad arithmetic. get_referrer_analytics ends its range one day before today because GA4 data lags, while get_search_console_metrics and get_bing_metrics end today.

How the workflow breaks down

1. Read connection state first. get_crawler_activity gives the connected flag and every observed bot with its first and last visit. A bot whose last visit predates your window did not crawl during it, whatever the page table says.

2. Rank pages by observed crawler activity. get_page_performance returns per-path occurrences with up to five leading bots per path, plus an explicit long-tail group. The row limit caps at 100, so compare rows returned against the total page-group count before reading absence as zero.

3. Add per-page technical state. get_site_diagnostics returns technical, content, AEO and performance scores per monitored page, open issue counts by severity and a last-audit timestamp, sorted worst first. It reads monitored pages only, so a crawled path that was never registered never appears.

4. Add citation and referral evidence. get_page_presence carries citation status and counts per page, get_source_urls recovers exact cited URLs, and get_referrer_analytics returns AI-referred sessions with their top landing pages.

5. Join and route. Match on a normalised host-and-path key and record absence precisely: not returned, not observed, explicitly not cited, or not audited.

What a run actually returns

A real site: 263 monitored pages, 13 cited, 250 not, 4 paths cited that were never in the monitored set. Across the same window the citation graph held 3629 distinct source domains, of which the brand's own accounted for 54 citations.

Read as a funnel that says 5 percent conversion, which would be wrong: the 250 uncited pages include policy pages and keyword indexes nobody would quote, so no denominator here means anything. The row carrying information is the 4. Engines quoted paths the monitoring set never registered, every one of them a page nobody on your side is watching, and finding those is what the join is for.

What to do with the output

One gate, then one routing rule.

The gate: if any connected flag comes back false, the backlog has exactly one item, connecting that integration. An empty table from a disconnected surface is not a finding. Two absences under it are not zeros either: a page missing from the top landing pages fell outside a truncated list, and one missing from the crawler table may sit in the long-tail group. Never divide citations by crawler occurrences: one comes from an aggregate with no period filter, the other from a rolling window.

The routing rule then applies to every page with crawler occurrences and no citation. Stop at the first match. Absent from get_site_diagnostics entirely means it is not a monitored page: register it and re-run, since you hold an unmanaged URL, not a diagnosis. A null last-audit timestamp with null scores means it was never audited: queue the audit and stop. One or more open CRITICAL issues makes it an engineering ticket, not a writer's brief. Scores present with no CRITICAL issue open makes it a content and authority question owned by editorial.

The Complete Prompt

I want a page-level AI observability audit. If I monitor more than one brand, ask which to use. Keep crawler requests, technical state, citations and AI referral traffic as four separate layers, never a funnel. Use the last 30 days. Ask me for my canonical host and URL aliases first, since no tool knows which hosts are mine.

1. Run get_crawler_activity. Report the connected flag and each bot with its first and last visit. Call it an all-time view with no visit counts and no status codes, and flag any bot whose last visit predates the window.
2. Run get_page_performance for 30 days. Report the connected flag, the returned date range, page rows with occurrences and leading bots, and the long-tail group. Compare rows returned against the total page-group count and flag truncation. If connected but returning no page rows, say so rather than inferring zero crawling.
3. Run get_site_diagnostics. Return each monitored page with its four scores, issue counts by severity and last-audit timestamp. Treat a null score as not audited, never as zero.
4. Run get_page_presence, then get_source_urls where an exact cited URL is needed. Use NOT_CITED only when the tool returns it, and never divide citations by crawler occurrences.
5. Run get_referrer_analytics for 30 days, plus get_search_console_metrics and get_bing_metrics by page if connected. Print every returned date range and note the GA4 range ends a day earlier.

If any connected flag is false, stop and return one item: connect that integration.

Otherwise build the matrix on canonical host plus normalised path, then route every crawled page with no citation, stopping at the first match. Absent from site diagnostics means register it as a monitored page. Null last-audit and null scores means queue the audit. An open CRITICAL issue means an engineering ticket. Otherwise it is editorial work. Give each row its evidence and owner.