What does an AI assistant read on your website?

I build websites and I run The Answer Report, an audit that fetches a page the way those crawlers do and hands back the text they received, and its free check settles most of this for your own site. Here is what it looks at, in crawl order.

The door: robots.txt and the named agents

The first thing a well-behaved crawler asks for is `/robots.txt`. The agents have names, and their owners publish them.

OpenAI documents four: GPTBot, which crawls content that may train its foundation models; OAI-SearchBot, which surfaces sites in ChatGPT's search features; ChatGPT-User, which fetches on a user's request; and OAI-AdsBot. Anthropic documents three: ClaudeBot, Claude-SearchBot and Claude-User. Perplexity documents two: PerplexityBot, which surfaces and links sites in its results, and Perplexity-User, which fetches on a user's request. OpenAI and Perplexity both document that robots.txt may not apply to those user-initiated fetches; Anthropic makes no such exception and says its bots honor the file.

Two things follow. Training and answering are separate jobs done by separate agents, so a blanket block is blunt. And Google's Google-Extended is not a crawler at all: it is a control token covering both training for future Gemini models and grounding, the material handed to Gemini at the moment somebody asks. Google states it does not affect inclusion or ranking in Search, so blocking it costs nothing in Search and takes your pages out of Gemini's grounding.

My own stance, mine and not a rule: I allow every agent that can put the work in front of a person, the training crawlers included. My site is a portfolio and a writing hub, so being inside a model is reach rather than a loss. A shop with a paid catalog, or a publisher whose archive is the asset, has a real reason to answer differently. What I would not do is let a default decide it. A hosting or CDN layer can ship a robots policy that turns assistant traffic away without anybody choosing it, a pattern The Answer Report keeps finding on live sites.

A minimal allow stanza:

User-agent: OAI-SearchBot
Allow: /

User-agent: ClaudeBot
Allow: /

User-agent: PerplexityBot
Allow: /

User-agent: Google-Extended
Allow: /

User-agent: *
Allow: /

Sitemap: https://example.com/sitemap.xml

Robots.txt is advisory, though: a front door, not a lock.

What it reads: the HTML your server returned

Google renders. Its documented pipeline is crawl, render, index, and in the render step a headless Chromium runs the page and executes the JavaScript. Most AI crawlers do not have that machinery.

Vercel and MERJ analyzed AI crawler traffic in server logs and reported that the OpenAI, Anthropic, Meta, ByteDance and Perplexity crawlers do not render JavaScript. They request JavaScript files. They never run them. The exceptions they found are Gemini, which rides on Googlebot's infrastructure, and AppleBot. SEO.AI describes ChatGPT's browsing tool the same way: "It retrieves the raw HTML. It doesn't run JavaScript or wait for scripts."

If your service list, your city or your hours arrive in the browser after a client-side fetch, they are not in the document the assistant read. The page looks finished to you because your browser did the work. The crawler did not.

The same logic covers the rest. An image is not text. A photograph of your hours or a services graphic gives a machine the alt text you wrote, or nothing. A scanned PDF is an image in a wrapper. If a fact matters, it exists as characters in the HTML or it does not exist.

It reads in chunks, not top to bottom

Dan Petrovic tested this against OpenAI's web search tool. What the model is handed is not a page: it is a small object with the title, the URL and a snippet of one to three sentences. When it wants more it asks for a larger slice of the same page, centered on a line number, and it can ask again at a different line. He calls it a sliding-window browsing pattern.

That changes where facts belong. If the plain statement of what you do, who you do it for and where you work sits three scrolls down under an About heading, you are betting the window lands there. Put it in the first screen, in a sentence a stranger could lift and quote. Ten years of writing prep lists in kitchens taught me the same lesson: what is not written where the next person looks does not exist.

Structured data: the same facts, labeled

Everything above is an assistant reading prose and inferring. Structured data skips the inference: JSON-LD is a block of labeled JSON in a script tag stating, in a vocabulary machines already agree on, what the page is about. The types that carry a small business are ordinary: Organization or LocalBusiness for the entity, Person for a practitioner brand, Service for what you sell, FAQPage for a question block genuinely on the page.

{
  "@context": "https://schema.org",
  "@type": "LocalBusiness",
  "name": "Example Plumbing",
  "url": "https://example.com/",
  "telephone": "+31 30 123 4567",
  "address": {
    "@type": "PostalAddress",
    "streetAddress": "Voorbeeldstraat 1",
    "addressLocality": "Utrecht",
    "addressCountry": "NL"
  }
}

Three rules I hold to. One graph, one identity: one block describing the business, not a second name and a third phone number across the site. The same facts as the visible page: if the schema says you open on Saturday and the page does not, you have published two versions of one fact. No invented fields to pad a score: an aggregate rating with no reviews behind it, a service you do not offer, an FAQPage for questions that appear nowhere. A claim the page does not support is worse than no claim.

Contact details have to be characters

The report I run scores contactability as its own category, separate from legibility, because a page can be readable and still not tell a machine how to reach the business.

Keep your form. A form is a place to type, not a fact to read. If an assistant is asked how to contact you and the only reachable thing is a submit button, the honest answer is that it could not find your details. Write the phone number, the email and the address as text, and let the tel and mailto links wrap characters already there.

llms.txt, in two sentences

llms.txt is a plain text file at the root of your domain that describes the business in your own words and points at the pages worth reading. It is a convention rather than a ratified standard, it does not override robots.txt, and The Answer Report's guide on what llms.txt is and whether you need one covers the rest.

How to see what they see

You do not need a tool for the first check. Fetch the page with a plain HTTP client and read what comes back.

curl -sL https://example.com/ | less

That is close to what a non-rendering crawler receives. Search it for what a customer would ask about: your city, your phone number, your main service, the sentence that says what you do. If you cannot find them there, an assistant cannot either.

I have watched these agents walk sites I built. They ask for robots.txt, they ask for pages, they take the text that came back, and they leave. That much is visible in any access log.

What this is not

This is not traditional SEO under a new name and it is not a ranking trick. It is legibility: making sure the true things about your business exist in a form a machine can pick up and repeat. A site can be beautiful, fast, well written, and still unreadable to the thing your customer asked.

It is also a different question from why an assistant fails to recommend you at all. Legibility is necessary and not sufficient; The Answer Report's guide on why ChatGPT doesn't recommend your business covers that side.

If you would rather see the machine's view of your own site than a description of it, the free Answer Report check fetches your homepage the way these crawlers do and shows you what came back.