A client asked me to start a conversion optimisation programme this week. Monthly commitment, analyst and developer, the full arrangement.
He said yes before I had finished explaining it. I told him to wait.
The Number That Decides It
His site does around 14,000 visitors a month.
I do not usually begin structured A/B testing below roughly 50,000. That is not a rule I invented to sound rigorous. It comes out of how the arithmetic behaves.
An A/B test splits your traffic in two and compares the halves. To say with any confidence that one version beat the other, you need enough people in each half that the difference cannot reasonably be explained by chance. At 50,000 visitors a month you can usually reach that point in two or three weeks. At 14,000 the same test needs considerably longer, and the longer it runs the more other things change underneath it. Seasonality. A price change. A campaign that brought in a different kind of visitor.
So you either wait months for one answer, or you stop early and pretend.
Most people stop early. Two weeks in, one version is ahead, and it is very tempting to call it. But an experiment that has not reached significance is not a result. It is noise with a chart attached.
What Usually Happens Instead
The commercial answer here is obvious. Take the money, run the tests anyway, and report on whatever moves. Sessions. Bounce rate. A dashboard with arrows pointing in the right direction. The client cannot easily tell the difference between a real finding and a plausible one, which is precisely the problem.
I did not want to be in that position in six months, explaining why the numbers had not moved.
So I told him what I actually thought. Then I sent him to Evidoo, a database of ecommerce A/B tests that records how often each pattern has been tested and how often it won. It is free to register for. I showed him where to find the patterns with a strong record across many tests, and told him to hand whatever he liked to his own developers.
He can implement most of it without me. He simply will not be able to prove which parts worked, and I said that clearly rather than leaving it to be discovered later.
Why This Is Not Generosity
I want to be transparent about the self-interest here, because a decision that costs nothing is not really a decision.
If I take on a testing programme that cannot conclude, one of two things happens. Either I invent enough activity to look busy, which I am not willing to do, or the client works out after several months that nothing measurable happened. The second outcome ends the relationship and the first one deserves to.
Turning it down costs me one month of work. Taking it costs me the client.
There is a version of this business built on getting to yes. It is not the one I run. We have never locked an ecommerce client into a long agreement, which means every month has to be worth paying for on its own merits, and that changes what you are willing to sell in the first place.
What I Told Him to Do Instead
Fix the things that do not need proving.
There is a category of change where the evidence is already overwhelming from other people’s testing. Making the add to cart button reachable without scrolling on mobile. Telling returning visitors they still have something in their basket. Showing a shopper how close they are to free delivery. These have been tested by thousands of stores and they win the large majority of the time.
You do not need your own experiment to justify those. You need a developer and an afternoon.
Structured testing is for the decisions where the answer genuinely is not obvious, and where you have enough traffic to find out. Come back when you do.
When Testing Does Pay
For contrast, here is what this looks like when the traffic supports it.
We recently added a free delivery progress bar to a client mini cart. Twelve days, 25,850 visitors, conversion up 35.74% and revenue per visitor up 66.41%, at 99.75% confidence.
That last figure is the whole point. It is what separates a result from a coincidence, and it is only available to you if you have the traffic to earn it. The full write-up of that test is on the MageCloud site.
Related Reading
Why Budget Does Not Decide Who I Work With — the same filter applied to who I take on in the first place.
Speaking at the UK SEO Summit 2026 — on running AI content workflows and where the human review has to sit.