Before you diagnose a conversion decline, check whether it is real

Suppose a business says conversion has fallen from 4% to 3%. That sounds like a quarter of its conversion performance has disappeared. The natural response is to ask what went wrong: the page, the offer, the price or checkout.

Sometimes that question arrives too early.

Before looking for the cause of a decline, I want to know whether the number measures the same thing in both periods, whether the people being compared are sufficiently alike, and whether anything commercially important has actually deteriorated.

Those are not questions for an elaborate audit. They are what keep a team from spending weeks explaining a problem that exists mainly in a report.

First check what was counted

Start with the numbers behind the percentage: what counted as a purchase, and what counted as an opportunity to buy?

Did a completed payment count in both periods? Did a tracking change alter the recorded event? Are refunds, delayed purchases, duplicate contacts or manually processed orders handled consistently? Were there comparable opportunities to buy?

If purchases have stopped being recorded after a checkout update, the reported rate might fall while actual purchases remain unchanged. That is an instrumentation problem, not evidence that customers suddenly dislike the offer.

Even accurate events can be divided by the wrong population. A visitor who arrived for a free article, a past customer seeking support and somebody evaluating a substantial purchase may have different reasons for being there. Their visits can belong in a useful site-wide report without becoming interchangeable evidence about lost suitable buyers.

A different audience can produce the same symptom

Imagine a deliberately simplified business with two groups. Its ready-to-buy visitors purchase at 5%, while earlier-stage visitors purchase at 1%.

In one period, 800 of 1,000 visitors come from the first group and 200 from the second. That produces 42 purchases, or 4.2%.

In the next period, the business attracts 5,000 visitors: 1,000 from the first group and 4,000 from the second. Their respective purchase rates are unchanged. The overall rate nevertheless falls to 1.8%, while total purchases rise to 90.

Nothing in those figures shows that the buying journey got worse for either group. The audience mix changed. That still leaves commercial questions: what did the additional traffic cost, how valuable were its customers, and could the business serve them well? More purchases do not automatically mean more profit.

The example is hypothetical, but the mistake is common: treating an aggregate movement as though it described a deterioration among comparable people.

A recorded difference is not necessarily a changed process

A rate can also move without a new underlying cause. A short launch or a small number of sales can produce quite large percentage swings when a handful of purchases fall into a different week.

If twenty people buy from 500 comparable opportunities one month and fifteen buy from 500 the next, the recorded rate has genuinely moved from 4% to 3%. The five-purchase difference is real. It does not, on its own, show that the sales process became worse.

I want to know how unusual the movement is against the variation we should reasonably expect, whether the pattern persists, and how much economic consequence is at stake.

Where the data are suitable, statistical process control can help distinguish ordinary variation from a process shift. Experiments are more useful for testing whether a proposed change caused an effect. A formal significance threshold, however, cannot decide whether the result matters commercially, and failure to detect a change is not evidence that performance is good enough.

A stable rate can still be a bad result

There is a danger in overcorrecting. If the rate has been consistently inadequate for years, the absence of a new decline is no reason to leave it alone.

A business might be consistently acquiring too few suitable customers, losing money on each sale or failing to help buyers reach a valuable first use. Its performance can be stable and still demand attention.

Nor does a verified defect need to wait for statistical confirmation. If a payment option reliably rejects eligible orders and the fix is safe and straightforward, repair it and check that it works. That observation justifies the narrow fix, not a confident claim about how much additional revenue the repair created.

Decide whether the difference matters

Conversion is an intermediate measure. The business ultimately cares about suitable customers, sustainable economics and the value it can deliver.

A lower rate can accompany more valuable demand. A higher rate can be harmful if it creates wrong-fit purchases, overwhelms a limited service or makes customers worse off. Some non-purchase is also the outcome of a sound decision.

Before starting a causal diagnosis, I try to answer four practical questions: Are we counting the same event? Are we comparing the right people and opportunities? Does the movement suggest more than ordinary variation? And is the possible loss important enough to change what we do?

Those checks tell me whether a comparable, consequential decline deserves causal diagnosis, or whether the real next move is narrower: repair the measurement, investigate a particular group, fix an obvious defect, address persistently poor performance, collect one more useful observation or leave the headline rate alone.

Diagnosis is valuable when it explains a real decision-relevant problem. Establishing that problem is itself part of the work.