When A/B testing is the wrong tool
We once had two possible names for a course and no particularly good reason to trust our own preference between them.
There was a related free event coming up, so we used its promotion to get some evidence. We split the first promotional email between the two event names while sending both groups to the same registration page. One version got more registrations from that email, and we used that result when choosing the eventual course name.
It was a good use of an experiment because the uncertainty was narrow, randomisation was cheap, the result arrived quickly and a real decision was waiting. It also mattered that we did not claim more than the test had shown. We learned which framing produced more registrations in that audience and context. We did not learn which course name would create more sales.
That is close to my ideal A/B test. The method was simple and the result was closely matched to the decision.
A test spends something even when the software is cheap
A/B testing is often discussed as though the main constraint is sample size. In the businesses I work with, the opportunity cost matters too.
A professional training launch might have only a few hundred genuinely plausible buyers in an intake. The launch window is short, many of the same people have seen previous offers and messages, and human sales or support interactions can affect what happens after the website.
We regularly make substantial changes to pages, offers and journeys between launches without trying to A/B test each one. Small intakes make clean tests difficult, but there is a commercial reason as well: the prospects and the launch window are scarce opportunities.
Using a large part of one intake to answer a marginal headline or CTA question means giving up the chance to use the same traffic to learn something more important. I think of the relevant prospects as an experimental budget. You can spend them on a question, but you do not get the untouched opportunity back.
That budget also has an expiry date. Much of our audience arrives at a launch having seen earlier prices, arguments and offers. Formal experimentation research treats carryover and interference as threats to interpretation. Recurring launches create a related practical problem for us: response in one intake can depend on what people saw before it.
The surrounding system can move as well. Sales conversations differ, payments are handled manually, exceptions are made and offers change between cohorts. By the time a low-volume result becomes convincing, the version of the business it describes may already have changed.
A precise answer can still be low-value
Controlled-experiment research distinguishes statistical significance from practical significance. A measured difference can be real and still too small to matter commercially. An underpowered, inconclusive result also does not show that there is no useful effect.
We have skipped tests where every plausible result would have led us to roughly the same action. I have also continued with a change after an inconclusive result when the test was weak and other evidence was strong enough to justify the decision.
I do not calculate a formal expected value of information for every test. The underlying question is useful enough: how valuable would it be to reduce this uncertainty, and what will it cost us to learn?
If A winning, B winning and no meaningful difference all leave the business doing essentially the same thing, the experiment may be technically possible and commercially pointless.
Sometimes the evidence is already strong enough to act. If an important link is broken or qualified prospects repeatedly hit the same obvious obstacle, delaying the fix solely to gain experimental certainty can be worse than acting.
The metric can answer the wrong question
Our naming test is a good example of another limit. The observable outcome was registration, so that is what the experiment told us about.
Registration mattered because we were choosing a name for an event and needed a timely decision. Had the real question been which course name produced better students or more revenue, the same test would have been weak evidence.
We have seen early metrics improve without establishing anything about sales, buyer quality or downstream value. We have also seen an aggregate result look less convincing once the more relevant audience segment was isolated.
Good experiment design tries to use short-term measures that are meaningfully connected to the outcome the organisation actually cares about. In a complex buying journey, that connection is often the difficult part: the easy signal might be a click or registration while the important result arrives much later.
I therefore keep the claim attached to what we actually measured. A registration test is evidence about registration. A click test is evidence about clicks. Moving from there to revenue, fit or long-term customer value requires another argument.
Some experiments cost trust as well as traffic
Some questions are technically testable and still not worth testing. We have avoided materially different price or payment-term tests among comparable prospects when the likely learning was not worth the fairness, customer-comparison or trust cost. I would make the same call on a deliberately misleading variant or one I already believed would create a worse customer experience.
Sometimes we change several connected parts of a journey together. If the real decision is which complete version to ship, knowing that one whole journey works better can be enough, even though it tells me less about which component caused the difference.
When I want the A/B test
The naming test worked because the experiment and the decision had almost the same shape: one narrow uncertainty, enough relevant exposure, a quick observable result and a real choice waiting on the answer. That is the kind of question I am happy to spend traffic on.
With thinner traffic, slower outcomes, weak proxies, contaminated groups or meaningful trust costs, I usually combine the evidence already available and make a proportionate decision. Customer conversations, support patterns, transaction data, behaviour elsewhere in the journey and previous launches cannot give randomisation's causal strength, but they may be more useful than spending a whole launch on a result that arrives too late.
In a low-volume business, I want each experiment to justify the customer opportunity it consumes. The scarce resource is the next clean chance to learn from a real customer.