CX Optimization Agency Message Testing Blog Search Start 7-day trial for $1 Upskilling Teams Community Resources Help

top-statistics-for-ab-testing

How to Deal with Outliers in Your Data

One thing many people forget when dealing with data: outliers.

Even in a controlled online A/B test, your data set may be skewed by extremities. How do you deal with them? Do you trim them out, or is there another way?

Keep reading

One-Tailed vs. Two-Tailed Tests (Does It Matter?)

One-tailed tests allow for the possibility of an effect in one direction. Two-tailed tests test for the possibility of an effect in two directions—positive and negative.

Simple as that concept may seem, there’s a lot of controversy around one-tailed vs. two-tailed testing. Articles like this one lambaste the shortcomings of one-tailed testing, saying that “unsophisticated users love them.”

On the flip side, some articles and discussions take a more balanced approach and say there’s a time and a place for both.

Let’s set the record straight.

Keep reading

A/B Testing Statistics: An Easy-to-Understand Guide

Testing tools are getting more sophisticated. Blogs are brimming with “inspiring” case studies. Experimentation is becoming more and more common for marketers. Statistical know-how, however, lags behind.

This post is filled with clear explanations of A/B testing statistics from top CRO experts. A/B testing statistics aren’t that complicated—but they are that essential to running tests correctly.

Here’s what we’ll cover (feel free to jump ahead):

  1. Mean, variance, and sampling;
  2. Statistical significance;
  3. P-values;
  4. Statistical power;
  5. Confidence intervals and margin of errors;
  6. Regression to the mean;
  7. Segmenting;
  8. Confounding variables and external factors.

And just in case you’re uncertain about why A/B testing statistics are so essential…

Keep reading

fishing net

Years ago, when I first started split-testing, I thought every test was worth running. It didn’t matter if it was changing a button color or a headline—I wanted to run that test.

My enthusiastic, yet misguided, belief was that I simply needed to find aspects to optimize, set up the tool, and start the test. After that, I thought, it was just a matter of awaiting the infamous 95% statistical significance.

I was wrong.

Keep reading

What’s on my mind

Hi, I'm Peep Laja—founder of CXL. I'm a former champion of optimization and experimentation turned business builder.

I do a lot of thinking, reading, and writing around business, strategy, and optimization. I send a weekly newsletter with what's on my mind on this stuff.

Subscribe

Categories