LLM SEO: how a model decides what to cite

// In this article+

LLM SEO is the practice of shaping a page to survive three separate steps inside a language model's answer pipeline: retrieval, chunking, and citation. Search Engine Land, Aleyda Solis, and most of the labs building these systems call the broader discipline generative engine optimization (GEO), answer engine optimization (AEO), or AI search optimization. Those three names cover one shift, from ranking a page to citing a passage. This piece works one level below that naming question, at the mechanism itself.

Each of the three steps runs on a different, documented process a site owner can actually change. Crawler permissions decide whether a page is even a candidate. Chunk structure decides which passage a model ever sees. Citation is computed separately from both.

What is LLM SEO, and what is actually being optimized?

None of the three steps behaves like classic ranking. A search engine scores a page and returns it in a list. A language model does not return a list. It assembles an answer out of passages pulled from many pages. It decides which passages earned a place in the text, and only then decides which of those get named as a source. A page can dominate a search results page and still contribute nothing to the answer, because it never survived the earlier two steps.

I started in search in the mid-1990s, when one crawler and one ranking algorithm covered the whole job. These three steps split what used to be one decision into three, and each one rewards a different kind of work.

Retrieval decides who is in the room. A page has to be crawled, indexed by the system doing the answering, and pulled as a candidate for a specific question before anything else about it matters.

Chunking decides which part of the page the model reads. A retrieval system does not hand a whole page to the model. It hands back whichever chunk scored closest to the question, and a page with one useful sentence buried in an unrelated paragraph can lose that chunk entirely.

Citation decides whether the model says where the answer came from. A passage can enter the context window and still go unnamed in the final text, because citation is computed after retrieval, not as a byproduct of it.

This is a narrower argument than the GEO vs SEO: what changes when the engine answers piece on this site, which explains why a top ranking stopped predicting a citation.

How does a model retrieve candidate content before it writes an answer?

Query fan-out is the technique that turns one question into several. Search Engine Land's guide to query fan-out ↗︎ describes the process in five stages: decomposition, expansion, execution, synthesis, and contextual results generation. A system breaks a query into sub-questions, expands each one, runs them as separate searches, and stitches the results into a context-aware answer. One search for the best CRM for a ten-person agency can spawn sub-queries on pricing, integrations, and alternatives. Each one pulls its own set of candidate pages.

Aleyda Solis's piece on Google's query fan-out technique ↗︎ frames the practical response as answering facets rather than head terms. She defines fan-out as an information retrieval technique that "expands a single user query into multiple sub-queries to capture different possible user intents". A page written only to the head term misses every sub-query that never surfaces in a rank tracker. No person ever typed those sub-queries. The model generated them.

That is also why retrieval returns passages rather than pages. Retrieval-augmented generation, or RAG, is the architecture behind most of this. It fetches relevant text at answer time instead of relying only on training data. Research on RAG best practices ↗︎ treats the retrieval workflow as a chain of choices, each weighed for how it balances performance against efficiency. A page written as one long argument, with the answer buried in paragraph nine, is optimized for a reader, not a retrieval system. That system never sees paragraph nine unless it wins its own contest against paragraphs one through eight.

A fan-out checklist for one page. List the three to five sub-questions a reader would actually ask around the target term. Give each one its own heading and its own direct answer in the first sentence under that heading. Skip the sub-questions with no real search behavior behind them: fan-out expands intent, it does not invent it.

How does chunking decide which part of a page a model ever sees?

A chunk is the unit of text a retrieval system actually embeds, indexes, and can hand back. It might be a sentence, a paragraph, or a short group of paragraphs, depending on how the system was built. A page is never one unit to a retrieval system. It is however many chunks the system cut it into, and each chunk is scored and retrieved on its own.

Mike King, founder of iPullRank, put that claim to a direct test. His team took a paragraph covering two unrelated topics, machine learning and data privacy. They split it into two single-topic paragraphs and re-embedded both versions. The machine-learning passage's cosine-similarity relevance score improved by 19.24% once it stopped sharing a paragraph with an unrelated topic, per iPullRank's write-up on the chunking test ↗︎. Nothing about the sentence itself changed. Only its company did.

Anthropic's engineering post on contextual retrieval ↗︎ documented the opposite failure, and a fix for it. Splitting a document into chunks strips out the surrounding context a sentence needs to make sense on its own. A chunk that reads "the company's revenue grew by 3% over the previous quarter" is nearly useless once retrieved, because it names neither the company nor the quarter. Anthropic's fix, contextual retrieval, prepends a short, chunk-specific context header before the chunk is embedded. Combining contextual embeddings with contextual BM25, a keyword-matching algorithm, cut the top-20-chunk retrieval failure rate by 49%, from 5.7% to 2.9%. Adding a reranking step cut the failure rate 67% against the same 5.7% baseline, to 1.9%.

A chunking checklist. One idea per paragraph, so a retrieval system never has to choose which topic a paragraph is actually about. A descriptive subhead above each section, since the subhead is often what gets prepended as context. No orphaned pronouns at a section boundary: a paragraph that opens "it reduced costs by 12%" reads as complete to a person who just finished the paragraph before it, and as meaningless to a chunk retrieved on its own.

Once a passage is retrieved, how does a model decide whether to cite it?

Anthropic's Citations documentation ↗︎ shows the citation mechanism plainly. In Claude's Citations feature, document contents are chunked to define the minimum granularity of possible citations. Sentence chunking, for example, "lets Claude cite a single sentence or chain together multiple consecutive sentences". The chunk boundary set earlier in the pipeline caps how precisely a model can ever point back at a source. A page chunked at the paragraph level cannot produce a sentence-level citation later, regardless of how well the sentence itself is written.

Research published since this piece first ran pushed the question further, asking whether citation is just a byproduct of retrieval rank or its own computation. A University of Amsterdam paper on citation mechanics ↗︎ offers what it calls "the first mechanistic account of how a large language model decides whether to attach an inline citation". The answer runs against the simple version of that question. Citation is not one isolated, readable signal. The paper maps it as a distributed, multi-stage "attributional ensemble" of attention heads and MLP layers, not a single, localized component.

That is black box territory. The division of labor across those layers is not something an outside operator can inspect, and I would not sell a single tactic here as certain. Theoretically, a passage that states its subject plainly still gives the mechanism more to anchor to, though the paper does not measure that link directly.

Put the two sources together: retrieval decides whether a passage enters the context window, and citation, computed afterward by a mechanism nobody outside the lab can fully inspect, decides whether the model names where it came from. Writing that names its subject plainly still helps, since a passage has to survive on its own once pulled from a page. "Acme's onboarding flow cut signup time to four minutes" carries an entity, a product, and a number to cite. "This cut signup time significantly" carries nothing anyone could cite with confidence.

How do the crawlers that feed this pipeline actually treat a site?

Every step above assumes a page was crawled in the first place, and each lab runs more than one crawler with a different job. Treating "block AI" as a single robots.txt decision is the most common operator mistake in the whole pipeline.

OpenAI's bots documentation ↗︎ states plainly that these settings are independent: a webmaster "can allow OAI-SearchBot in order to appear in search results while disallowing GPTBot to indicate that crawled content should not be used for training". Blocking GPTBot alone protects training data. It does nothing to citation eligibility in ChatGPT's search product, which runs on OAI-SearchBot.

Anthropic's crawler help center article ↗︎ draws the same three-way split. ClaudeBot trains models, and Claude-User fetches on a live user's behalf. Claude-SearchBot is the third job: it "navigates the web to improve search result quality for users. It analyzes online content specifically to enhance the relevance and accuracy of search responses". Perplexity's crawler documentation ↗︎ splits the same two jobs into two bots: PerplexityBot "is designed to surface and link websites in search results on Perplexity" and "is not used to crawl content for AI foundation models". Perplexity-User handles the live, per-query fetch.

A robots.txt check that takes ten minutes. Pull the current robots.txt file. Confirm OAI-SearchBot, Claude-SearchBot, and PerplexityBot are allowed, since blocking any of them removes a site from that engine's citation pool entirely. Decide separately, and deliberately, whether GPTBot and ClaudeBot are allowed, since that choice is about training data, not about whether a page can appear in a live answer.

What to do this week

The five checks below cover all three steps at once, the kind of structural review an outside AI SEO services engagement usually runs before touching a single word of copy.

1. Audit robots.txt against the crawler table above so OAI-SearchBot, Claude-SearchBot, and PerplexityBot are not accidentally blocked alongside their training-only counterparts, per OpenAI's bots documentation ↗︎ and Anthropic's crawler help center article ↗︎.
2. Run the iPullRank single-idea test ↗︎ on the highest-traffic pages: find any paragraph carrying two topics or two entities and split it into two.
3. Add a one-sentence descriptive lead-in to each major section so a chunk pulled out on its own still names its subject, the same job Anthropic's context header ↗︎ does at the embedding stage.
4. Submit changed URLs through IndexNow, a protocol under which "search engines adopting the IndexNow protocol agree that submitted URLs will be automatically shared with all other participating search engines" (IndexNow documentation ↗︎), rather than waiting for a normal recrawl cycle.
5. If building on the Anthropic API or evaluating Claude directly, run the Citations feature ↗︎ against a page's own content and read which sentences actually get selected at that granularity.

FAQ

Is LLM SEO different from GEO or AEO?

Not as a rival category. GEO is the broader discipline covered at the category level in the GEO vs SEO article on this site: earning a citation instead of a rank. LLM SEO, as used here, is the mechanism underneath that discipline, the retrieval, chunking, and citation steps that decide whether any of it works on a given page.

Does blocking GPTBot stop a page from being cited in ChatGPT?

No. GPTBot and OAI-SearchBot are independent settings in OpenAI's own documentation: disallowing GPTBot opts a page out of training data, while OAI-SearchBot is the crawler that indexes a page for citation inside ChatGPT's search product. Blocking the wrong bot is the most common version of this mistake.

Do chunk size and structure matter more than backlinks for AI citation?

Not on a single readable signal. The University of Amsterdam's interpretability research on citation decisions found the mechanism to be a distributed ensemble across attention heads and MLP layers, not a single, localized component. What is documented is that citation granularity is capped by chunk structure, not by links, per Anthropic's own Citations documentation and its contextual retrieval work. Link-based SEO still decides whether a page gets retrieved into the candidate pool at all, before chunking and citation ever run.