Getting cited when the buyer asks an AI

A growing share of buyers never reach your site. They ask an assistant, read the answer, and act on whichever three products it named. You cannot buy a place in that answer, and there is no dashboard for it — but the inputs are ordinary, checkable, and mostly things a competent engineering team can ship in an afternoon. This is what we do to our own site, and what we ask a client's team to do to theirs.

Two different bots, and confusing them costs you the answer

The vendors run more than one crawler, and they do different jobs. Reading them as one thing is the most common mistake we find in a robots.txt:

• Training crawlers collect text to train on. Blocking one does not remove you from the product; it removes you from the next model. OpenAI's GPTBot is this: "Disallowing GPTBot indicates a site's content should not be used in training generative AI foundation models."
• Search crawlers fetch pages so the assistant can cite them, right now, in an answer. Blocking one takes you out of answers while leaving you in the training set — the opposite of what most teams believe they are choosing. OpenAI is explicit: "Sites that are opted out of OAI-SearchBot will not be shown in ChatGPT search answers."
• User-triggered fetchers run when somebody pastes your URL into a chat — and two of the three vendors say these may not obey robots.txt at all. OpenAI on ChatGPT-User: "Because these actions are initiated by a user, robots.txt rules may not apply." Perplexity on Perplexity-User: "Since a user requested the fetch, this fetcher generally ignores robots.txt rules." Anthropic makes no such carve-out: "Anthropic's Bots respect 'do not crawl' signals by honoring industry standard directives in robots.txt."

The names, as they appear in a user-agent line: GPTBot, OAI-SearchBot and ChatGPT-User for OpenAI; ClaudeBot, Claude-SearchBot and Claude-User for Anthropic; PerplexityBot and Perplexity-User for Perplexity; and CCBot for Common Crawl, which many smaller models are trained on.

Two more names are not crawlers at all, and confusing them with the ones above is the second most common mistake. `Google-Extended` controls whether pages Google has already crawled may be used for Gemini training and grounding, and Google states plainly that it "does not impact a site's inclusion in Google Search nor is it used as a ranking signal". `Applebot-Extended` is the same shape: "Applebot-Extended does not crawl webpages. Webpages that disallow Applebot-Extended can still be included in search results."

Decide each one deliberately and write the decision down. A company that sells being cited by assistants and is silent about whether they may read it has not made a decision; it has an accident. Allow about a day for a change to take effect — OpenAI documents roughly 24 hours from a robots.txt edit, and Perplexity says the same.

What we do on this site

Our robots.txt names each of those agents in its own group and allows it. That repetition is not redundancy: a bot that finds a group with its own name ignores the * group completely, so an inherited rule is not inherited at all.

The signed-in product is a different matter. It is not disallowed in robots.txt — it answers with a noindex header instead, because a crawler forbidden to fetch a page never learns that it should forget it. Blocking and de-indexing are opposite instructions, and reaching for the wrong one is how a private area stays in an index for a year.

The page has to contain its own words

Most of what makes a page quotable is unglamorous: the words have to be in the HTML.

A site rendered entirely in the browser hands a fetcher an empty <div> and a script tag. Search engines will sometimes run that script. Link previews, most crawlers and the fetchers behind assistants usually do not — so a page that reads perfectly in Chrome is, to them, a page with nothing on it. Our own public pages are rendered to HTML at build time for exactly this reason, which is checkable from outside: fetch one and read what comes back, or run curl over it with an assistant's user-agent and see whether there is a sentence in the response.

Google says the same thing from the other side, and it is the most useful sentence published on this subject: to be eligible for AI Overviews or AI Mode, "a page must be indexed and eligible to be shown in Google Search with a snippet". There is no separate AI optimisation to buy — "There are no additional requirements to appear in AI Overviews or AI Mode, nor other special optimizations necessary."

The rest is ordinary technical hygiene that also happens to decide what an answer engine can attribute:

• A title and a description that describe the page rather than the company.
• One h1 that says the subject, and h2s a reader could skim as a summary.
• A canonical link, so the same page at two addresses does not split its own evidence.
• Schema.org data that repeats only what the visible page says — an Organization, an FAQPage where the page really is questions and answers, an Article with a date and an author.
• An address that does not exist answering 404 rather than the front page, because a fetcher that gets a 200 for everything learns nothing about which of your pages are real.

llms.txt: worth writing, not worth believing in

llms.txt is a proposal for a plain-text file at your root listing your important pages with one line each, so a model reading your site has a map. We publish one. Then the caveats, which the people selling llms.txt audits leave out.

It is a proposal, by one author, and its own site says so: the specification is "open for community input" and the page describes itself as an "informal overview". There is no standards body behind it. More to the point, we could not find a single AI vendor that documents reading one — not OpenAI, Anthropic, Perplexity, Google, Apple or Common Crawl. Every crawler page above is silent about it. The AI labs publishing llms.txt files for their own documentation is often cited as adoption; publishing a file is not the same as consuming one.

Google, meanwhile, says the opposite in as many words: "You don't need to create new machine readable files, AI text files, or markup to appear in these features."

Write it anyway. It costs one build step and it doubles as a page inventory a human can read, and if it is ever adopted you are already in it. Just do not let it displace the work above, and do not let anyone charge you for it as a ranking factor.

What gets quoted

Across the pages we watch, the passages that end up quoted have the same shape:

• A direct answer in the first two sentences of a section, under a heading that asks the question.
• A specific number or limit, with the source or the method beside it.
• A named trade-off — where the thing does not work, said plainly.
• Text, not an image of text. A screenshot of a pricing table is invisible to every reader in this article.

The inverse is also consistent. A page that opens with two paragraphs of throat-clearing before it says what the product does gets paraphrased from somebody else's description of you, and the assistant cites them.

Measuring it

There is no click to attribute, so the question changes from "how much traffic" to "are we named, and by whom".

1. 1Write the twenty questions a buyer would actually ask, in their words, not yours.
2. 2Ask them across the four engines, on a schedule, from a clean session.
3. 3Record whether you were named, in what position, and which page was cited.
4. 4Watch the cited page, not just the brand: it tells you which of your pages the engines trust.

Fresh sessions matter — an engine that has been talking to you about your own product all week will name it in a way it never would for a stranger. We track this weekly for clients as part of the AI visibility work, and it is the reason that work is sold against agreed terms rather than against impressions.

How we measure everything else is the other half of this, for the channels where there is at least a thread to point at.