Don’t Do A/B Testing If You Aren’t Ready [Rant]

A/B testing is highly useful, no question here. But a lot of businesses should not be doing it. They’re not ready yet.

Roughly speaking, if you have less than 1000 transactions (purchases, signups, leads etc) per month – you’re gonna be better off putting your effort in other stuff.

A lot of microbusinesses, startups and small businesses just don’t have that transaction volume (yet).

You might be able to run A/B tests with just 500 transactions per month too (read: how many conversions do I need?), but you need bigger impacts per successful experiment to improve the validity of those tests.

If it’s like 240 vs 260 conversions (~8% uplift), you actually don’t have enough evidence to know one way or another. I highly recommend reading up on A/B testing statistics to know why that is.

You also have to keep in mind the cost of optimization. I don’t just mean the cost of tools you need for a testing program (which can be free), but the cost of the time spent:

• figuring out what to test – using a data-driven approach instead of guessing randomly (which would render the whole testing program useless),
• designing the treatment (provided that you’re testing non-trivial stuff),
• setting up the test in the testing tool (using css and javascript/jquery),
• doing QA on the tests.

Now let’s say you get that 8% lift, and it’s a valid winner. You had 125 leads per week, and now you have 135 / week. Is the ROI there? Probably not.

So when you calculate your needed sample sizes before you run the test, do the math on the ROI as well. What would be the value of X% lift in actual dollars?

What about microconversions?

Yeah you could be measuring clicks on buttons and stuff, but there is no causal relationship between higher microconversions and higher revenue. You can entice more people to click by over promising or setting false expectations. Or just shift the problem onto the next page – where they have to pay, but they don’t.

So if you’re spending all that energy running tests on microconversions – and don’t actually know for sure what the impact on the bottom line is – you’re better off doing other stuff. Like making your product better. Figuring out your unit economics, so you could acquire customers better. Building an audience.

A/B testing does not equal optimization – you should still optimize

Testing is a great way to validate hypotheses. But even if you don’t have the numbers to have a meaningful amount of quantifiable data, you can still do heuristic analysis, usability evaluation and all the qualitative stuff:

• talk to your customers,
• run user tests,
• survey your clients,
• poll your website visitors.

(Here’s more on how to optimize low traffic websites).

Conclusion

Time is a precious resource. It might be better spent elsewhere than A/B testing when you’re still small – because math.