How to diagnose a conversion problem before you optimise it

Steve New

I recently reviewed the sales journey for a substantial professional training programme. At first glance, there was plenty wrong with it.

Some prospects were still asking basic questions about participation and fit, there had been checkout and discount problems, and human follow-up occasionally failed. Onboarding generated support noise. A few people requested refunds shortly after the first live session. The sales material itself was long and had accumulated information over time.

You could make a reasonable case for changing almost any part of the journey. That is exactly why the work needed diagnosis rather than a longer improvement list.

A good conversion audit can include serious quantitative and qualitative research, causal hypotheses and sensible prioritisation. The mistake is treating the discovery of things that are weak, broken or improvable as sufficient evidence for what caused the commercial result and what should change.

Diagnosis is the reasoning that connects those observations to explanations, tests the alternatives and determines what action the evidence can support.

When the cause of a conversion problem is unclear, diagnosis has to come before optimisation.

This example comes from work I did inside the training organisation. I've removed identifying details. The useful question was narrower than “What could we improve?”

Where does a plausibly suitable prospect's progress first materially weaken between serious consideration and successfully starting the programme, why does it weaken, and what should change before the next sales cycle?

By plausibly suitable, I mean somebody whose goals, background and apparent ability to participate made them a credible fit independently of whether they eventually bought.

By the end of the analysis, the answer was much smaller than the original list of problems suggested. The main repeated weakness in the journey I examined was not checkout or onboarding. It appeared when people who could already see what the programme offered had to decide whether joining was genuinely right and workable for them now. Some of that difficulty was avoidable, while other hesitation reflected real constraints that a good sales journey should preserve rather than try to overcome. Several obvious improvement projects turned out not to be justified at all.

None of the individual ingredients here is groundbreaking – systematic CRO research already combines quantitative and qualitative evidence. Experience mapping starts by defining scope and intended use. Good customer research connects investigation to a business need. Causal-inference work forces alternative explanations into view. Decision analysis asks whether reducing an uncertainty is worth the cost of doing so. Controlled experimentation gives us stronger ways to estimate intervention effects when the conditions support it.[1–6]

What matters to me is how those ideas connect in an ambiguous commercial journey. The business decision determines which uncertainties matter. The evidence narrows the explanations that could change that decision. The consequence of being wrong determines how much confidence we need before acting.

What does the evidence support changing now?

Start with the decision you are trying to improve

“Conversion is weak” describes a result. It does not tell you what decision the business actually needs to make.

The real question might be whether to change the price, rebuild a sales page, improve acquisition, alter an offer, repair checkout, rethink onboarding, or simply stop spending time on an area that is already working well enough. Those decisions require different evidence.

If the concern is that conversion has fallen, I first check whether there is a comparable, commercially meaningful decline to explain. Are the numerator and denominator defined consistently? Has tracking changed? Has the audience mix shifted? Is the movement greater than ordinary variation would lead us to expect? A stable rate can still be poor, and a reproducibly broken checkout should be repaired without waiting for a statistical exercise. The point is to identify the real question before inventing its cause.

There are also two different decisions in the journey. The business decision is what the organisation needs to decide or change, while the customer decision is what the buyer is trying to work out.

In the training case, the business needed to know what to change before the next sales cycle. The customer was deciding whether a demanding six-month professional programme was sufficiently valuable, appropriate and practical for them to join now.

Once those decisions are clear, the relevant population becomes clearer as well. A visitor who spends thirty seconds on the page is not equivalent evidence to a prospect who attends an event, reads the offer, asks specific questions, considers the commitment and then decides not to join. Depending on the question, both may matter. They should not automatically be treated as the same kind of loss.

The same issue appeared in the onboarding denominator. It initially looked messy. There were support questions, access problems and inconsistent records, which made it easy to assume that buyers were getting lost after purchase.

Before interpreting those anecdotes, I reconstructed who was actually eligible to attend the first live Opening. The records contained duplicates, corrected identities, late purchasers and other cases that made the neatest initial population unreliable.

After reconciliation, 53 of 54 eligible participants attended at least part of the Opening. That made widespread failure to reach the first session a poor basis for a major onboarding overhaul. Individual access and calendar defects still needed attention. The count did not establish later persistence, full-course completion or anything about prospects who never bought.

Without the right population, even a precise percentage can support the wrong diagnosis. The question determines which evidence matters, not simply the data that happens to be available.

Keep the observation separate from the explanation

Suppose twenty people start checkout and do not finish. “Twenty people abandoned checkout” is an observation. “Checkout friction is causing lost sales” is an explanation.

Or suppose several prospects mention price. “Several prospects said the programme was expensive” is an observation. “The programme is overpriced” is an explanation.

The explanations may eventually prove useful – the mistake is behaving as though they arrived directly from the data. The same observation can fit several explanations that would lead to very different actions. This is the same reason I treat disagreement between customer words and behaviour as diagnostic evidence rather than a vote between two sources.

Checkout abandonment might reflect a technical problem, unexpected price, loss of trust, a payment constraint, an unresolved fit question or somebody simply changing their mind.

A price objection might reflect genuine inability to pay, cash-flow timing, low priority, uncertainty about value, a poor comparison with alternatives or an entirely sensible decision that the programme is not worth the trade-off for that person.

I do not try to preserve every explanation that could conceivably be true. That becomes analysis for its own sake. An explanation deserves to stay alive when it is plausible and when believing it rather than another live explanation would materially change what we do.

If several subtle explanations would all lead to the same cheap clarification on the same page, separating them may have little decision value. If one means lowering the price, another repairing checkout, another narrowing acquisition and another leaving the journey alone because the loss is healthy qualification, the distinction matters.

Several explanations can also be true at once, so diagnosis does not require one grand winner. It requires enough separation to know which mechanisms deserve which responses.

Look for evidence that separates the explanations

The training case began with several explanations that could all have sounded convincing in a meeting.

Perhaps prospects did not understand the programme's value, or perhaps the price was too high. Checkout had produced real problems, onboarding looked noisy, and early refunds raised the possibility that some people only discovered a mismatch after the programme began.

What mattered was evidence capable of changing the relative credibility of those explanations, rather than simply supporting one of them.

Missing evidence only weakens an explanation when the source could reasonably have revealed it. A checkout record shows an exit, not its motive. A support inbox shows the questions that reached somebody, not the ones that never reached anyone.

Value was not simply a comprehension problem

Many serious prospects could describe what they valued in the programme: live practice, direct feedback, professional application, certification and the chance to develop practical capability. That weakened a simple explanation that they were failing to progress because they did not understand the offer.

Understanding the offer was not the same as finding its value sufficient relative to price, time or alternatives. But the evidence narrowed the issue enough to make a sweeping value-proposition rewrite less attractive as the first project.

Price was real, but not one thing

There were genuine affordability cases. At least one plausibly suitable prospect could join once a workable payment arrangement was found, so financial constraints were clearly real in some purchases. But the evidence did not show that standard price explained most serious non-purchase.

“Price matters to some people” and “the standard price is the main conversion constraint” are different claims with very different consequences. The first may justify payment options, scholarships or another route for particular customers. The second could justify changing the economics and positioning of every sale. The evidence supported the first much more strongly than the second.

Checkout contained defects without becoming the whole diagnosis

There had been genuine checkout and discount problems, and those deserved repair. What was missing was evidence that the standard checkout systematically prevented otherwise-ready buyers from purchasing. Fix the demonstrated defect – don't inflate its causal scope.

A broken payment path does not need a lengthy research project before repair, but its existence does not establish that checkout explains the wider conversion problem.

Onboarding looked worse before the denominator was fixed

Onboarding produced plenty of support noise, from access and calendar questions to inconsistent internal records. Those symptoms could easily have prompted a major project. The reconciled attendance count made widespread failure to reach the Opening much less plausible, while leaving genuine local faults to repair. That is the kind of evidence I find most useful: it changes which actions still make sense.

Early refunds pointed to different explanations

At least three refund requests followed the first live experience, but their reasons were mixed. One participant valued the work yet could not make the necessary commitment alongside another major obligation. Another attended, decided the programme was not right for them and used the refund option as intended. An early refund may reveal avoidable expectation-setting problems or a legitimate correction of fit. The evidence supported tracking the reasons consistently, not immediately redesigning the curriculum.

Find the earliest defensible point where progress weakens

Once the main explanations have been narrowed, I want to know the earliest point the evidence supports where a relevant customer's ability or willingness to make useful progress materially deteriorates. I call that the first meaningful weakening. By meaningful, I mean there is evidence that the difficulty is repeatedly affecting whether relevant customers can reach a sound next decision. Questions or normal consideration friction do not meet that threshold on their own.

The first meaningful weakening is not necessarily the first measurable funnel drop, the biggest percentage loss or the first thing to fix. A problem can begin upstream while a separate downstream defect deserves immediate attention, and more than one constraint can matter. Intervention priority depends on consequence, prevalence, confidence, changeability and the downside of getting the action wrong. The concept locates the diagnostic chain rather than commanding us to optimise its earliest point.

It also helps to distinguish three locations that are often collapsed into one. A contributing mechanism may begin in one place, customer progress may materially weaken later, and the visible symptom may appear later again. That is why a visible conversion problem can look bigger, later or more distributed than the underlying constraint.

A prospect might abandon at checkout while the important uncertainty began when an earlier page created a misleading expectation about price, workload or fit. Checkout is where the failure becomes visible. It does not follow that checkout created it.

In the training journey, the evidence pointed to a relatively narrow decision stage. Plausibly suitable prospects often understood what the programme offered, but still had to work out whether it was right and workable for them now.

Questions repeatedly involved experience level, workload, live attendance, certification, flexibility, professional fit, payment and who could give a reliable answer.

The evidence went beyond repeated questions. It included deferrals, at least one purchase that became possible when a workable payment arrangement was found, buying conversations delayed or lost through failed follow-up, and important participation conditions surfacing only after serious consideration or purchase. Human support often resolved unusual circumstances, but it could also become fragile. One serious prospect wanted to speak with someone before committing, and missed follow-up and crossed wires became part of the buying experience. Clear ownership of questions would address that operational weakness without removing the human help that worked.

The earliest defensible weakening was therefore the move from:

“This looks valuable”

to:

“This is right and workable for me now.”

The evidence located a stage of decision-making, not one timestamp shared by every prospect. That was enough to guide the next actions.

There is another question before calling that weakening a problem: should this customer actually progress? Some people faced real constraints around workload, timing, family commitments or money. Better marketing could not and should not remove all of them. One interested prospect could make the live sessions but deferred because the wider practicalities did not fit that intake.

Some friction also performed a useful qualification function. Participation requirements made the commitment more visible before purchase. A minimum financial contribution could act partly as a screen rather than simply an obstacle to maximise away. In one case, a prospect remained willing to buy while the pre-purchase interaction itself gave the business enough new information to decide that the longer relationship probably should not begin.

Calling something “healthy qualification” needs evidence – otherwise it risks becoming a flattering explanation for poor conversion. A requirement may protect commitment, teaching quality or capacity while also excluding someone who could have succeeded. Before treating the screen as effective, ask what it protects, who might be excluded and whether a workable, fairer route would preserve its purpose.

The loss should make sense in relation to real requirements, downstream fit, customer success or the kind of relationship the business is trying to create. This is why I treat non-purchase as something to classify before trying to eliminate it.

With that safeguard, I find four possibilities useful: avoidable loss, healthy qualification, an external constraint and genuinely unknown.

For a consequential purchase, the useful goal is not always the maximum number of immediate yeses. A good journey should help a suitable customer reach a sound decision, even when the answer is later or no.

Conversion diagnosis narrows observations and competing explanations through discriminating evidence to supported explanations, the first meaningful weakening, a decision threshold and proportionate action.

As evidence accumulates, some explanations strengthen, some weaken, and new evidence can reframe the diagnosis. The five outcomes are equally legitimate when supported by the evidence.

Conversion diagnosis narrows observations and competing explanations through discriminating evidence to supported explanations, the first meaningful weakening, a decision threshold and proportionate action.
As evidence accumulates, some explanations strengthen, some weaken, and new evidence can reframe the diagnosis. The five outcomes are equally legitimate when supported by the evidence.

Decide whether the diagnosis is good enough to act on

A diagnostic can almost always be made more detailed, but the useful question is whether more detail is worth acquiring.

By the end of the training analysis, I was more confident about the repeated weakening than about its precise distribution across price, workload, timing, schedule and fit. More research might have narrowed those shares, but was unlikely to change the first actions. The recommendations were already stable: make fit and feasibility easier to judge, maintain one reliable version of the current commercial and participation rules, and make serious buying questions much harder to lose in human follow-up.

Before stopping, I challenge the emerging diagnosis: What is the strongest remaining alternative? What evidence would change my mind? Have I explained away inconvenient evidence too easily? Without that challenge, the stopping rule becomes an excuse to settle on a preferred answer.

My practical rule is to stop when the next action is stable enough that further investigation is unlikely to change it enough to justify the cost, delay or attention required. That is a practical application of the same idea formalised in decision analysis as the value of information: the value of learning depends on both the chance that current uncertainty could lead us to the wrong decision and the cost of being wrong.[4]

Two questions make the judgement easier: What decision could the next piece of evidence actually change? And how bad would it be if we acted now and were wrong?

Suppose the evidence supports rewriting one paragraph that repeatedly leaves serious prospects unsure about participation. The change is cheap, reversible and produces feedback quickly, so it rarely needs a high degree of confidence before trying it.

A broad price cut, site rebuild, repositioning or offer redesign is different. It affects more customers, consumes more resources, can damage what works and may take months to evaluate. The evidence burden should rise with the consequences of being wrong.

The intervention should not be bigger than the diagnosis. That is not a preference for small changes in every situation. An unsupported platform, incoherent offer architecture or fault that cannot economically be patched may justify substantial work. A large intervention is proportionate when the diagnosis supports its scope and consequences, not when the improvement wish list is long. A website rebuild is a large bundled hypothesis until the evidence supports that choice.

I would keep the internal rules and follow-up repairs separate where they could be done independently. But if a serious prospect needs an answer about participation, a current rule and someone responsible for responding have to come together. That is a reason to coordinate the two changes, not a reason to launch one large bundled project.

Decide what follows

Once a finding is strong enough to matter, there are several legitimate dispositions. A single diagnostic can produce several at once.

FIX

The problem is sufficiently established and the proportionate repair is clear. If a payment button reliably fails, suitable buyers cannot proceed and the repair is straightforward, fix it.

The same applies to known contradictory information, a stale rule or another established problem whose repair does not depend on resolving a larger causal question first.

TEST

An important uncertainty remains, and deliberately changing something is a worthwhile way to learn about its effect.

Testing makes most sense when the alternatives are real, enough relevant exposure exists, the result can be observed in time to matter and different outcomes would actually change the decision. Controlled experiments can provide much stronger causal evidence than ordinary before-and-after observation, but only if the experiment and its metrics answer the commercial question that matters.[6] In low-volume journeys, that is also why A/B testing can be the wrong tool even when it is technically possible.

The fact that something can be tested is not enough – a precise answer to a low-value question can still be a poor use of scarce customer attention.

MEASURE

A missing observation could materially change the action, and collecting it is worth doing before changing the underlying system. In the next training cycle, useful measurement included recording why serious prospects defer or decline, separating genuine checkout failures from ordinary abandonment, tracking whether important questions receive a closed-loop answer, and capturing simple reasons for early refunds.

The point is not instrumentation for its own sake. It is obtaining a particular observation because that observation can resolve a live decision.

NOT YET

The issue may matter, but the contemplated action is not responsibly supported yet. Several live explanations may still imply different large interventions. The relevant population may be impossible to reconstruct. The next natural cohort may produce much better evidence without forcing an artificial research exercise now.

NOT YET means the evidence is not sufficient to act and the decision remains open. It can coexist with MEASURE for the same decision: defer a broad price cut while recording the reasons serious nonbuyers declined or waited.

LEAVE ALONE

Current evidence supports leaving this unchanged.

The apparent problem may be healthy qualification, the area may already work adequately, or the constraint may be external. The downside of changing it may exceed the likely benefit, or another constraint may simply deserve attention first.

That is different from simply not having enough evidence yet.

MEASURE means learn this now. NOT YET means do not make this decision yet. LEAVE ALONE means do not change this on the current evidence.

The training diagnostic produced several dispositions at once.

Specific defects deserved repair, while some unanswered questions needed better measurement. Cutting the standard price was NOT YET: affordability mattered in some cases, but the evidence did not show that standard price explained most serious non-purchase. The evidence also did not justify rebuilding checkout, overhauling onboarding, redesigning the curriculum, substantially increasing sales email or trying to eliminate every early refund.

Those non-actions were part of the value of the work.

For the larger recommendations, the diagnosis also had to remain revisable. I would change my view if better evidence showed that most plausibly suitable prospects actually failed to understand the offer, price overwhelmingly accounted for serious non-purchase, standard checkout failures were widespread, the first live experience repeatedly created the mismatch, or an earlier constraint consistently appeared before the fit-and-feasibility decision.

A useful diagnosis should be firm enough to act on and specific enough to be wrong.

Follow the intervention far enough to see whether it helped

The three recommendations aimed to make fit and feasibility easier to judge before purchase and serious questions easier to resolve. They were proposals, not observed results, and their success should not be judged simply by whether the immediate purchase rate rises.

I would also want to know whether serious prospects reach decisions with fewer repetitive unresolved questions, whether fewer important conditions are discovered only after payment, whether suitable buyers still reach the first live experience successfully, and whether the pattern of early fit-correction refunds improves without suppressing useful qualification.

A local conversion improvement matters only insofar as it helps the customer and business outcome the diagnosis was supposed to improve. Experimentation practice makes the same point through the choice of evaluation criteria: short-term metrics are useful when they provide a credible signal about the longer-term objective rather than merely being easy to move.[6] This is also why I treat checkout as a useful event rather than the automatic commercial endpoint.

There is no need to track every downstream metric. Follow the result far enough that a local improvement cannot hide a worse outcome elsewhere, then update the diagnosis from what happened.

Verifying a repair is different from proving its commercial effect. If a broken discount code works after correction, the technical repair is verified. Claiming that it generated a particular amount of additional revenue would require a credible comparison with what would otherwise have happened. The fix can be justified without making a larger causal claim about its financial effect.[5][6]

The professional training analysis did not attribute every non-purchase to price, workload, timing or fit, or uncover a single root cause behind every symptom. It did something more useful: made several expensive explanations less credible, identified where customer decisions repeatedly became harder, distinguished avoidable uncertainty from legitimate constraints, supported proportionate changes and gave us good reasons not to start several larger projects.

The business decision determined which uncertainty mattered, the evidence narrowed the explanations, and the consequence of being wrong determined how far the investigation needed to go before action.

A good diagnosis reduces uncertainty until the next useful decision becomes clear.

Selected sources

  1. Peep Laja / CXL, “Conversion Research”, describing ResearchXL and quantitative and qualitative conversion research. https://cxl.com/conversion-optimization/conversion-research/
  2. Jim Kalbach, Mapping Experiences, 2nd edition, particularly framing, scope and focus for experience mapping. O'Reilly Media, 2020. https://www.oreilly.com/library/view/mapping-experiences-2nd/9781492076629/
  3. Steve Portigal, Interviewing Users: How to Uncover Compelling Insights, 2nd edition, especially Chapter 1, “Interviewing Addresses a Business Need”. Rosenfeld Media, 2023. https://rosenfeldmedia.com/interviewing-users-second-edition-sample-chapter/
  4. Douglas W. Hubbard, How to Measure Anything: Finding the Value of Intangibles in Business, particularly “Quantifying the Value of Information”. Wiley, 3rd edition, 2014. https://www.oreilly.com/library/view/how-to-measure/9781118836446/c07.xhtml
  5. Nick Huntington-Klein, The Effect: An Introduction to Research Design and Causality, research questions, alternative explanations and causal identification. Chapman & Hall/CRC, 2nd edition, 2025. https://www.routledge.com/The-Effect-An-Introduction-to-Research-Design-and-Causality/Huntington-Klein/p/book/9781032580227
  6. Ron Kohavi, Diane Tang and Ya Xu, Trustworthy Online Controlled Experiments: A Practical Guide to A/B Testing, particularly Chapter 7, “Metrics for Experimentation and the Overall Evaluation Criterion”. Cambridge University Press, 2020. https://www.cambridge.org/core/books/abs/trustworthy-online-controlled-experiments/metrics-for-experimentation-and-the-overall-evaluation-criterion/0A8BF7F06A1338FF6D10C49F4CD8A198