fbpx

A/B Testing

AB testing statistics

Testing tools are getting more sophisticated. Blogs are brimming with “inspiring” case studies. Experimentation is becoming more and more common for marketers. Statistical know-how, however, lags behind.

This post is filled with clear explanations of A/B testing statistics from top CRO experts. A/B testing statistics aren’t that complicated—but they are that essential to running tests correctly.

Here’s what we’ll cover (feel free to jump ahead):

  1. Mean, variance, and sampling;
  2. Statistical significance;
  3. P-values;
  4. Statistical power;
  5. Confidence intervals and margin of errors;
  6. Regression to the mean;
  7. Segmenting;
  8. Confounding variables and external factors.

And just in case you’re uncertain about why A/B testing statistics are so essential…

Keep reading

Top 20 A/B Ecommerce Test Ideas

There’s nothing that always works and pretty much nothing that never works either. Websites are highly contextual.

That being said, there are tests that tend to have a very high win rate. These are the test ideas that, while they don’t work 100% of the time, work more often than not.

Naturally, everything depends on the specific implementation — a good idea implemented poorly will not yield any results.

The following 20 testing ideas come from our own client-based research done over the years.

Keep reading

Predicting Winning A/B Tests Using Repeatable Patterns

If you ever ran a highly trustworthy and positive a/b test, chances are that you’ll remember it with an inclination to try it again in the future – rightfully so. Testing is hard work with many experiments failing or ending up insignificant. It’s optimal to try and exploit any existing knowledge for more successes and fewer failures. In our own practice we started doing just that.

Keep reading

How to Segment A/B Test Results to Find Gold

You run an A/B test, and it’s a winner. Or maybe it’s flat (no difference in performance between variations). Does it mean that the treatments that you tested didn’t resonate with anyone? Probably not.

If you target all visitors with the A/B test, it merely reports overall results – and ignores what happens in a portion of your traffic, in segments.

Keep reading

UX Research and A/B Testing

A/B testing is common practice and it can be a powerful optimization strategy when it’s used properly. We’ve written on it extensively. Plus, the Internet is full of “How We Increased Conversions by 1,000% with 1 Simple Change” style articles.

Unfortunately, there are experimentation flaws associated with A/B testing as well. Understanding those flaws and their implications is key to designing better, smarter A/B test variations.

Keep reading

Categories