Connect AI Crawler Observations, Citations, and Referral Traffic

Every page with crawler occurrences and no observed citation raises one question: whose problem is it, engineering's, editorial's, or nobody's yet. This workflow keeps four data layers apart, then answers that question per page with a rule that stops at the first match.

The GEO problem it solves

The compression happens twice: observed therefore processed, cited therefore clicked. get_crawler_activity is built from bounded bot state rather than retained HTTP events, and says so: each bot with first-seen and last-seen dates plus an all-time occurrence count, and an explicit note that raw events are unavailable. No status codes, no response bodies, no per-page rows, no period input. And an AI-referred session never names the citation behind the click.

The windows differ mechanically too, the quiet source of bad arithmetic. get_referrer_analytics ends its range one day before today because GA4 data lags, while get_search_console_metrics and get_bing_metrics end today.

How the workflow breaks down

1. Read connection state first. get_crawler_activity gives the connected flag and every observed bot with its first-seen date, last-seen date and all-time occurrence count. A bot whose last-seen date predates your window has no current-window observation in this layer, whatever the page table says.

2. Rank pages by observed crawler activity. get_page_performance returns per-path occurrences with up to five leading bots per path, plus an explicit long-tail group. The row limit caps at 100, so compare rows returned against the total page-group count before reading absence as zero.

3. Add per-page technical state. get_site_diagnostics returns technical, content, AEO and performance scores per monitored page, open issue counts by severity and a last-audit timestamp, sorted worst first. It reads monitored pages only, so a crawled path that was never registered never appears.

4. Add citation and referral evidence. get_page_presence carries citation status and counts per page, get_source_urls recovers exact cited URLs, and get_referrer_analytics returns AI-referred sessions with their top landing pages. Pass the same 30-day evidence window to both citation tools; the analytics range can still end one day earlier because of GA4 lag.

5. Join and route. Match on a normalised host-and-path key and record absence precisely: not returned, not observed, explicitly not cited, or not audited.

What a run actually returns

A valid run returns four separate populations: bot observations, monitored-page diagnostics, page citation evidence for the requested window, and AI-referred sessions for the analytics range. Their row counts can differ even when every call is healthy.

Reading those rows as a funnel would still be wrong. A monitored page can be uncited without being eligible for citation, and a referral session never identifies the citation that preceded it. The actionable row is a URL that is observed in one layer and absent from the next operational inventory, with that absence labelled precisely.

What to do with the output

One gate, then one routing rule.

The gate: if any connected flag comes back false, the backlog has exactly one item, connecting that integration. An empty table from a disconnected surface is not a finding. Two absences under it are not zeros either: a page missing from the top landing pages fell outside a truncated list, and one missing from the crawler table may sit in the long-tail group. Never divide citations by crawler occurrences even when their requested windows match: their populations and collection systems differ.

The routing rule then applies to every page with crawler occurrences and no citation. Stop at the first match. Absent from get_site_diagnostics entirely means it is not a monitored page: register it and re-run, since you hold an unmanaged URL, not a diagnosis. A null last-audit timestamp with null scores means it was never audited: queue the audit and stop. One or more open CRITICAL issues makes it an engineering ticket, not a writer's brief. Scores present with no CRITICAL issue open makes it a content and authority question owned by editorial.

The Complete Prompt

I want a page-level AI observability audit. If I monitor more than one brand, ask which to use. Keep crawler requests, technical state, citations and AI referral traffic as four separate layers, never a funnel. Use the last 30 days. Ask me for my canonical host and URL aliases first, since no tool knows which hosts are mine.

1. Run get_crawler_activity. Report the connected flag and each bot with its firstSeen, lastSeen and all-time occurrences. Call it an all-time bot-state view with no raw events or status codes, and flag any bot whose lastSeen predates the window.
2. Run get_page_performance for 30 days. Report the connected flag, the returned date range, page rows with occurrences and leading bots, and the long-tail group. Compare rows returned against the total page-group count and flag truncation. If connected but returning no page rows, say so rather than inferring zero crawling.
3. Run get_site_diagnostics. Return each monitored page with its four scores, issue counts by severity and last-audit timestamp. Treat a null score as not audited, never as zero.
4. Run get_page_presence with period 30, then get_source_urls with period 30 where an exact cited URL is needed. Preserve any topic, tag, provider or funnel scope on both calls. Use NOT_CITED only when the tool returns it, and never divide citations by crawler occurrences.
5. Run get_referrer_analytics for 30 days, plus get_search_console_metrics and get_bing_metrics by page if connected. Print every returned date range and note the GA4 range ends a day earlier.

If any connected flag is false, stop and return one item: connect that integration.

Otherwise build the matrix on canonical host plus normalised path, then route every crawled page with no citation, stopping at the first match. Absent from site diagnostics means register it as a monitored page. Null last-audit and null scores means queue the audit. An open CRITICAL issue means an engineering ticket. Otherwise it is editorial work. Give each row its evidence and owner.