Your page is fast for people and slow for crawlers

A browser is extraordinarily patient. It will wait for your fonts, retry a failed script, hold a half-painted page on screen for two seconds rather than show nothing, and hydrate whenever your bundle finally arrives. Performance work is largely the art of making that patience invisible.

An answer engine has none of it. It issues a request, takes what comes back, and moves to the next URL in the queue. If your server is still thinking, it leaves.

Two numbers, two different pages

We pointed our own speed checker at this site while building it. Two pages on the same host, same CDN, same deployment:

                          time to first byte      readable
                          cold      warm          text
/blog/llms-txt-explained  856ms     66-289ms      12%   (1020 words in 50KB)
/pricing                  1940ms    155-219ms      3%   (110 words in 33KB)

Both pages are fine in a browser. Nobody complained about the pricing page. It is smaller than the blog post, and warm it answers in under a fifth of a second. It is still the one an answer engine is most likely to abandon before reading, and least likely to find anything quotable in if it stays.

Two separate problems, and separating them is the point.

The first is the cold column. The pricing page renders per request, so the first fetch after a quiet period pays for a cold function: nearly two seconds, against about 180ms once it is warm. That gap matters more for crawlers than for people precisely because of who arrives. Your readers cluster — a link is shared, a campaign runs, the function stays warm. A crawler turns up on its own schedule, often to a page nothing else has requested in hours, which means it disproportionately gets the slow path. The number a monitoring dashboard shows you is the average your readers experienced. The number a crawler experienced is the tail.

The second is the readable column, and it does not vary at all. Three percent is structural: most of that document is a framework’s serialised data payload rather than words. No amount of warming fixes it, and no performance dashboard we had was reporting it.

Why the usual metrics miss this

Core Web Vitals describe an experience: how quickly the largest element paints, how much the layout jumps, how fast the page responds to a tap. They are good metrics and they are measuring a human being.

A crawler paints nothing, taps nothing, and experiences no layout shift. The parts of the request it is subject to are the parts those metrics deliberately abstract away — time to first byte, document size, and whether the content is in the response or arrives later. A page can score well on every vital and still hand a crawler an empty shell after a two-second wait.

This is not an argument against measuring vitals. It is an argument that they answer a question about people, and there is now a second audience with different constraints reading the same pages.

The three that matter

Time to first byte, separated from total time. The first is your server thinking: an uncached template, a query on the page, a cold function. The second adds the body coming down the wire. They have completely different fixes, and a single “load time” hides which one you have. On our cold pricing page it was 1940ms of thinking and 6ms of transfer, which rules out every fix involving bytes.

Redirects before anything is served. Each hop is a full round trip before a byte of content moves. One is normal. More than one usually means two rules half-solving the same problem, and a crawler working to a budget spends that budget on your redirect chain.

How much of the document is readable text. The number almost nothing reports, because for a browser it is irrelevant. Three percent means a crawler that does not execute JavaScript has 110 words to work with in 33 kilobytes of markup, and unlike a cold start it does not improve on the second visit.

What to do about the first one

Cache the HTML. Not the assets — those are almost certainly cached already — the document itself.

A slow time to first byte on a content page nearly always means the page is being generated per request when its content changes daily at most. A static render, or an edge cache with a short max-age and stale-while-revalidate, turns a variable number into a constant one regardless of what the page does underneath — and, more to the point, removes the cold path entirely, so the crawler arriving at three in the morning gets the same response as the reader arriving mid-campaign.

Measure it cold if you want the number a crawler sees. Hit the page once after a quiet period rather than five times in a row: the second request through the fifth are measuring your cache, which is a real number and not the one in question.

Watch for the header rather than the intent. Plenty of setups cache aggressively at the CDN and still send the document with Cache-Control: no-store, which instructs every intermediary to do the opposite of what was planned. Ours did.

The one that is not really about speed

The readable-content number is filed under performance because that is where it gets measured, but it is a visibility problem wearing a performance costume. A page that is three percent text is not slow. It is empty, to a reader that does not run your code, and the fix is not a cache — it is moving the content to the server.

It is also the failure that hides best, because every human test of it passes. The page looks right. The content is there. Your colleagues confirm it. The only way to see it is to read what arrives before anything runs, which almost nobody does, because for twenty years there was no reason to.