A dashboard measuring the wrong rivals is worse than no dashboard

We spent an afternoon inside a competing AI visibility product, analysing a project for a founder social network. The dashboard was complete. Scores, a competitor leaderboard, a heatmap, a prompt list, change indicators. Everything a buyer expects to see.

The brand at the top was a debate app. The nine competitors were other debate apps. The visibility score was 0%, reported with total confidence, on a trial with three days left to run.

Nothing was broken

That is the part worth sitting with. No error appeared. No panel was empty. The numbers were internally consistent and correctly computed — the brand genuinely was absent from answers about debate software, because it is a founder network. Every calculation in that product did exactly what it was built to do, against a set of inputs that described a different company.

The failure was upstream of every feature, in the one place nobody checks: the entities being measured. And the product had no way to express doubt about them, because a competitive set is not the kind of thing software usually asks twice about.

Why this happens rather than being a one-off

Competitive sets are configured once, during onboarding, at the moment the user knows least about the product and is most eager to get past the form. Discovery proposes some names, the user glances at them, and the set is fixed. From then on it is an input to every calculation and a subject of none.

Then it decays. Categories move: a company launches, another pivots, a third is acquired and stops being marketed. The set that was defensible in January stops describing the market by June. Nothing announces that, because every chart keeps rendering.

Discovery getting it wrong at the start is the dramatic version. Quiet decay is the common one, and it produces the same result more slowly.

The asymmetry that makes it dangerous

A tool with no data tells you it has no data. You look at an empty state, understand that you know nothing, and go and find out.

A tool with the wrong set hands you a complete, plausible picture. You read a 0% and conclude you have an urgent visibility problem. You brief an agency, commission a content programme, and spend a quarter competing for answers about a category you are not in. Every step is taken in good faith on a number that was never about you.

The confidence is what does the damage. A dashboard reporting a precise-looking figure carries an implicit claim that the question it answers is the question you asked.

How to check yours in five minutes

Open whichever tool you use and read the brand name at the top. Not the domain — the brand, as the product understands it. That single field is what every mention is matched against, and getting it wrong is more common than it sounds, particularly where a founder’s name, a product name and a legal entity all appear on the same site.

Then read the competitor list and ask two questions of each name. Would a buyer comparing us seriously consider this? Have I seen it in an actual answer? A name that fails both is dragging every average you look at.

Finally, ask what is missing. The most useful list in this kind of product is the brands the engines named that you are not tracking — it is free, because those answers were captured anyway, and it is the earliest warning that the category has changed shape. A company appearing in a quarter of your answers that nobody has made a decision about is either a competitor you have not admitted to or a false positive worth excluding on purpose. Undecided is the one state that helps nobody.

What a product owes you here

Three things, none expensive.

• Surface the untracked names it already saw, rather than silently discarding them because a plan limit capped the list.
• Ask periodically whether the set is still right, at a moment that is not onboarding. A prompt costs nothing and prevents the whole failure.
• Score every brand in the set the same way, including yours. A tool that measures your appearances carefully and approximates everyone’s else is not producing a comparison — and that error always flatters the customer, which is why it survives.

We are not writing this from a position of having always got it right. Our own comparison scoring used to read a rival’s position with a cruder proxy than it used for the customer’s brand, which meant the two sides of a side-by-side were not measured the same way. That is fixed, and it is exactly the class of bug that a leaderboard renders beautifully.