JavaScript and AI search: what a crawler sees before your code runs

There is a version of your site that you have almost certainly never looked at: the HTML your server sends, before a single line of your JavaScript runs. For a person it is a transitional state that lasts milliseconds. For most AI answer engines it is the whole page.

What actually happens when a crawler fetches you

A browser does a great deal of work on your behalf. It requests the document, parses it, discovers the scripts, fetches them, executes them, lets your framework hydrate, waits for the network calls that hydration triggers, and paints the result. By the time you look at the page, several seconds of coordinated effort have happened.

An answer engine fetching a page to cite it does one step of that. It requests the document and reads what came back. There is no execution stage in the common case, and where one exists it is a budget rather than a promise: Googlebot renders, on a queue, when it decides the page is worth the compute. The crawlers that feed assistants — OAI-SearchBot, PerplexityBot, ClaudeBot — are cheaper operations, closer to curl than to Chrome.

So the question is not whether your site works. It is what arrives in the first response, because for a growing share of the machines deciding whether to quote you, that response is the entire encounter.

The failure this produces

A client-rendered page serves a shell: a container element, a script bundle, and a serialised blob of data the framework will turn into content. Every human check of that page passes. It looks right in your browser, it looks right to your colleagues, it looks right in the screenshot you put in the deck. The content is unambiguously there.

Then an assistant is asked a question your page answers, fetches it, finds a div and a script tag, and quotes a competitor whose equivalent page arrived as text. Nothing errored. No monitoring fired. The only symptom is an absence somewhere you were not looking.

This is the most consequential difference between optimising for a ranked list and optimising for a generated answer. Google has spent a decade getting good at rendering JavaScript, and a site that relies on that has been fine. The engines composing answers have not made the same investment, and there is no particular reason to expect they will — rendering is expensive, and at their volumes the cheap path is sufficient for most of the web.

How to see what a crawler sees

The simplest check takes one command and no tooling:

curl -s https://example.com/your-page | wc -c
curl -s https://example.com/your-page | grep -c "a phrase from your page"

If the phrase is not in the response, it is not in the page as far as a non-rendering crawler is concerned. The byte count matters too: a large document with none of your words in it is the signature of a framework serialising its data payload into the markup, which is a different problem from a page that is merely empty.

Our website speed checker reports this as a percentage — how much of the document is readable text — for exactly this reason. It is the number no conventional performance tool reports, because for a browser it genuinely does not matter.

Disabling JavaScript in your browser is a reasonable second check, but it is not the same test. A browser with JavaScript off still applies your CSS, still follows noscript fallbacks, and will show you content that a crawler reading raw markup would not weight the same way. Read the response, not a rendering of it.

What to move to the server

Not everything, and framing it as all-or-nothing is why this problem persists. The question is narrower: which content do you want quoted?

• The substance of the page. The paragraphs that answer the question the page exists to answer. If a passage would be a good citation, it has to be in the first response.
• The facts a machine needs to place you. Your name, what you do, your pricing, your structured data. An assistant deciding whether this Acme is the Acme it saw elsewhere is working from what arrived.
• Navigation, in real links. A crawler discovers your other pages through href attributes in the markup. A menu built from a JavaScript array is a menu no crawler traverses.

What can stay on the client: interactivity, personalisation, anything behind authentication, dashboards, configurators, and the long tail of things nobody would cite. A pricing calculator can hydrate. The pricing table it is attached to should not.

The framework answers

Every major framework now has a server-rendering path, and the work is usually less than it sounds because it applies to a minority of your pages. In Next.js, a component is server-rendered unless it says otherwise, and the failure mode is a "use client" directive sitting further up the tree than anyone intended. In Nuxt and SvelteKit, the equivalent is a component that fetches on mount rather than on load. In a single-page app with no server story at all, prerendering the handful of pages you want cited is a smaller project than migrating the application.

A caution about the shortcut: serving different markup to crawlers than to people is cloaking, and search engines police it. Prerendering services that return a rendered snapshot to bots sit close enough to that line that it is worth reading their documentation carefully. The durable version is to serve everyone the content.

How to tell whether it worked

The mechanical check is the one above: your words are in the response. The outcome check is slower and more honest — whether engines start citing the page. Those are different timescales, and conflating them leads people to undo good work because nothing changed in a fortnight.

Server-side crawler logs are the fastest real signal, because they show the fetch itself rather than its consequences. A page that starts being fetched regularly by answer crawlers is a page that has entered the pool; whether it gets quoted then depends on whether the passage is quotable, which is a separate problem with a separate fix.

What this does not fix

Server-rendering a page nobody would cite produces a fast, legible page nobody cites. It removes a blocker; it does not create a reason. If your content is a paraphrase of what is already on ten other sites, a crawler reading it perfectly will still prefer the source. And if your robots.txt blocks the crawler, none of this has been read at all.