The technical foundation for getting cited by AI engines: crawler access, rendering, HTML quality, structured data, and discovery. A prioritized, run-anywhere checklist for any site.
Generative engines are the new discovery layer. When someone asks ChatGPT, Perplexity, Google's AI Overviews, or Gemini about your category, the model builds an answer from pages it can reach, read, and trust. If your site is hard to fetch or hard to parse, you lose before ranking even begins.
Most GEO advice fixates on content and authority. Both matter - but they sit on top of a technical foundation that decides whether an AI system can ingest your pages at all. Brilliant content buried behind client-side JavaScript, blocked crawlers, or broken markup never enters the answer set. You can't be cited for something the model never saw.
This is the hub checklist for the technical side of Generative Engine Optimization (GEO). It covers the plumbing: crawler access, rendering, HTML quality, structured data, discovery signals, and the emerging llms.txt convention.
By the end you'll have a concrete, prioritized checklist you can run against any site, plus pointers to deeper guides for each layer. Let's make your pages crawlable and citable.
Technical GEO shares a foundation with technical SEO, but the consumers are different - and less forgiving. Google's infrastructure has spent two decades learning to render JavaScript, follow messy redirects, and reconstruct intent from imperfect pages. Most AI crawlers are far more literal. Three differences reshape your priorities:
Multiple crawlers, multiple purposes. Instead of one dominant bot, you now face training crawlers, retrieval crawlers, and user-triggered fetchers from OpenAI, Anthropic, Perplexity, Google, and others. Each behaves differently and each can be allowed or blocked independently.
Extraction, not just ranking. Classic SEO asks "does this page rank?" GEO asks "can the model pull a clean, quotable passage from this page and attribute it to you?" That rewards structure and self-contained answers over keyword density.
Other Articles
Why AI Confuses Your Brand - Entity Disambiguation Fixes
AI answer engines often merge or misattribute similarly named brands. Learn why it happens, how to diagnose it across ChatGPT, Perplexity, and Gemini, and the entity disambiguation fixes that make your brand unmistakable.
Wikipedia & Wikidata for AI: How to Earn (and Keep) a Presence
AI engines lean on Wikipedia and Wikidata to describe brands and entities. Here's how notability and sourcing rules really work, and the policy-compliant way to earn and keep a presence.
Thinner rendering budgets. Several AI crawlers fetch raw HTML and do little or no JavaScript execution. Content that a modern browser assembles at runtime can be effectively invisible to them.
Get the foundation wrong and every downstream tactic - E-E-A-T, freshness, off-site citations - is wasted effort.
Use this table as your at-a-glance audit; the sections below expand each row. Work top to bottom - access problems make everything else moot.
# | Layer | What to verify | Deep-dive guide |
1 | Crawler access | AI user-agents aren't blocked in `robots.txt` or at the WAF/CDN | |
2 | Rendering | Core content exists in the raw HTML, not just after JS runs | |
Run a free audit: see if ChatGPT, Gemini and Copilot recommend you, in about a minute.
If the bots can't fetch your pages, nothing else on this list matters. The first audit is always access, and it has two failure points: your robots.txt and your edge (CDN/WAF) rules.
The major AI user-agents are documented publicly. OpenAI's crawler documentation separates GPTBot, OAI-SearchBot, and ChatGPT-User for training, search, and user-triggered fetches. Anthropic's crawler documentation covers ClaudeBot, Claude-SearchBot, and Claude-User, including how to allow or block each agent. Perplexity's crawler documentation covers PerplexityBot and Perplexity-User for indexing and user-initiated fetches. Google's crawler documentation defines Google-Extended as a control token for Gemini and Vertex AI training without affecting Search indexing.
The nuance most teams miss: blocking training crawlers is not free. For engines that retrieve live, being reachable is a precondition for being cited. Disallowing every AI bot to protect your content can also remove you from the answers your buyers see. Decide deliberately, agent by agent.
A minimal, permissive stance looks like this:
User-agent: GPTBot
Allow: /
User-agent: OAI-SearchBot
Allow: /
User-agent: ClaudeBot
Allow: /
User-agent: PerplexityBot
Allow: /
Sitemap: https://www.example.com/sitemap.xml
Then verify at the edge. Cloudflare, Akamai, and similar providers increasingly ship default AI-bot blocking or managed rules, so a bot allowed in robots.txt can still be 403'd upstream. Check real server logs, not just the file. The full agent-by-agent breakdown lives in robots.txt & AI Crawlers: Which Bots to Allow.
Assume the crawler sees your raw HTML and nothing more. Googlebot renders JavaScript in a second wave, but many AI crawlers do not - Vercel's analysis of AI crawler behavior found that the ChatGPT and Claude crawlers fetched JavaScript files but did not execute them. If your headline, body copy, or key facts only appear after a client-side framework hydrates, those crawlers get an empty shell. The fix is to serve meaningful HTML on the first response:
Server-side rendering (SSR) or static generation (SSG) for content pages, so the answer text is present in the initial payload.
Prerendering or dynamic rendering as a fallback for heavily client-rendered apps.
Progressive enhancement: the core content and links work without JS; interactivity layers on top.
To test what a bot actually sees, disable JavaScript in your browser, use curl to fetch the raw HTML, or check Google's URL Inspection "view crawled HTML." If your primary content is missing from that source, it's missing for AI too. The full playbook is in JavaScript Rendering & AI Crawlers: Why Your Content May Be Invisible.
Extraction quality tracks HTML quality. A model pulling a quotable passage does far better with a clear document than with a soup of nested <div>s. Give it structure to grab onto:
One descriptive <h1>, then a logical <h2>/<h3> hierarchy that mirrors the questions the page answers.
Real semantic elements - <article>, <section>, <table>, <ul> - instead of styled generic containers.
Answer-first passages: lead each section with a direct, self-contained statement a model can lift verbatim.
Descriptive alt text and captions so non-text content is still readable.
Speed and reliability matter too. Crawlers on a budget abandon slow responses, and intermittent 5xx errors or "soft 404s" (a 200 status on an empty or error page) can quietly drop pages from consideration. Serve fast, stable responses and correct status codes. Keep an eye on genuine 404s and redirect chains - every hop is a chance for a crawler to give up.
See your mentions across ChatGPT, Claude and Perplexity in real time, the moment buyers ask.
Structured data turns a page into facts a model can trust and attribute. Schema.org markup doesn't guarantee a citation, but it disambiguates who you are and what a page is - which is exactly what generative engines need to attribute a claim to the right entity. Prioritize the markup that establishes identity and content type:
Organization with sameAs links to your official profiles, tying your brand to a single, consistent entity.
Article / BlogPosting with a clear author, datePublished, and dateModified to signal freshness and authorship.
Person for author entities, connected via sameAs to reinforce expertise (E-E-A-T).
Breadcrumb to make site structure explicit.
Validate everything with Google's Rich Results Test and the Schema Markup Validator - invalid JSON-LD is worse than none. For the full entity-focused breakdown (and how it differs from AEO-specific FAQPage/HowTo markup), see JSON-LD Schema Guide for AI Citations.
Crawlers still need a map and a clean set of URLs. Three signals do most of the work:
robots.txt. This is the cheapest way to surface new and updated pages.Architecture is a citability lever, not just a crawl one: a well-linked, self-contained page is easier to retrieve and quote. Go deeper in Site Architecture for AI: Making Every Page Answerable.
llms.txt is a proposal, not an established ranking factor. The llms.txt convention suggests a Markdown file at your root that points models to your most important, clean content. It's low-cost and harmless to add.
Be honest about adoption, though: no major AI engine has publicly confirmed it uses llms.txt to rank or cite pages, and prominent voices in search have been skeptical of treating it as a must-have. Treat it as a low-effort experiment that may help discovery, not as a substitute for crawlable HTML and structured data. The evidence-based case is laid out in llms.txt Explained: Should You Add It?.
Work the checklist in order - access first, polish last. Here's a repeatable pass:
robots.txt, check that AI user-agents aren't disallowed, then check server logs and edge rules for silent 403s.curl. Is the core content in the raw HTML?<h1>, semantic structure, correct status codes, and fast, stable responses.sameAs entity links resolve.That last step is where measurement closes the loop. Qwairy tracks your visibility and citations across ChatGPT, Claude, Perplexity, Gemini, and Google AI Overviews, monitors AI crawler activity on your site, and surfaces which sources engines pull from - so you can tell whether a technical fix actually moved you into more answers. Fixing crawlability is only worth it if you can see the result; that's the measurement layer Qwairy provides.
Go deeper on the technical stack: Configure AI crawlers in robots.txt, test JavaScript rendering for AI crawlers, improve site architecture for AI retrieval, and treat llms.txt as an optional experiment rather than a ranking shortcut.
Technical GEO is unglamorous, and that's exactly why it's an edge. While competitors chase content volume, a site that AI crawlers can reach, render without JavaScript, parse as clean HTML, and understand as a coherent entity gets pulled into answers the others never enter. Run the six-layer checklist, fix access first, and measure whether each change earns you more citations. The foundation compounds - every satellite tactic works better once the plumbing is right.
Technical GEO is the set of technical practices that make a website reachable, readable, and interpretable by the crawlers and models behind generative engines like ChatGPT, Perplexity, and Google AI Overviews. It covers crawler access, rendering, HTML quality, structured data, and discovery signals - the foundation content and authority sit on top of.
They share a foundation, but GEO's consumers are stricter. Instead of one dominant search bot, you face many AI crawlers with different purposes, several of which don't execute JavaScript. GEO also optimizes for clean passage extraction and entity attribution, not just ranking, so structure and self-contained answers matter more.
Only deliberately. Blocking training crawlers may protect content, but for engines that retrieve live, being reachable is a precondition for being cited - so a blanket block can quietly remove you from AI answers your buyers see. Decide agent by agent rather than disallowing everything.
Often not. Googlebot renders JavaScript in a second wave, but several AI crawlers fetch raw HTML and do little or no JS execution, so client-side-rendered content can be invisible to them. Serving server-rendered or static HTML with your core content in the first response is the safe default.
No. Schema markup doesn't force a citation, but it disambiguates your identity and your content type, which helps engines attribute claims to the right entity. Prioritize Organization, Article, Author, and sameAs markup, and always validate it - invalid JSON-LD can do more harm than none.
Track two things: crawler activity on your site (which bots hit which pages) and your actual presence in AI answers (citations and share of voice across engines). A platform like Qwairy monitors both, so you can connect a technical change to a measurable shift in how often engines cite you.
Track your mentions across ChatGPT, Claude, Perplexity and all major AI platforms. Join 1,500+ brands monitoring their AI presence in real-time.
Free trial • No credit card required • Complete platform access
3 |
HTML & speed |
Clean semantic markup, fast responses, no soft errors |
This guide |
4 | Structured data | Organization, Article, Author, `sameAs` entity markup |
5 | Discovery | XML sitemaps, internal links, correct canonicals |
6 | `llms.txt` | Optional guide file - with realistic expectations |