# Become great at **statistics for A/B testing**

### Avoid costly testing mistakes stemming from misuse and misunderstanding of statistic

If you’re not fluent in A/B testing statistics, you won’t be able to tell whether your tests suck.

#### Online course

By **Georgi Georgiev**, Owner of Web Focus LLC

**Course length:** 3h 30min

Some of the companies that train their teams at CXL Institute:

A lot of your “winning” tests are probably not winners at all. Learn to call bullshit when needed, and be the person who advocates proper scientific approach in your team.

## In 10 sessions, you’ll learn

- How to run A/B tests with a sound statistical design in a variety of scenarios – multiple test variants, multiple outcomes, non-binomial data, and others.
- How to align the statistical design with the questions at hand to get business insights while avoiding common mistakes.
- The logic behind statistical hypothesis testing and concepts like statistical significance, confidence intervals, statistical power, and others.

## Introduction video (4min)

## Understand the complexities involved in planning and evaluating A/B tests

Avoid costly testing mistakes stemming from misuse and misunderstanding of statistics, and improve the ROI of all your A/B testing efforts, with Georgi Georgiev’s guidance.

*Anyone interested in genuinely understanding the math behind CRO and A/B testing absolutely has to use CXL Institute. My whole team has a great time discussing the material and aligning our views on processes we use, and to come up with ideas to implement in the future.*

## This course is right for you if…

- You can’t define statistical significance correctly without looking it up on Google.
- Your A/B tests produce a lot of “winners,” but your clients aren’t seeing improvements.
- You’re planning and analyzing A/B tests, but you don’t understand the statistical underpinnings of the testing process.
- You’re not confident in the outcomes of your tests and are unsure how much trust to put in them
- You have an in-house statistical tool you want to improve, or you use a third-party A/B testing software you want to understand better

## This course is probably not for you if…

- You are just starting with CRO and have little to no practical experience with A/B testing.
- You don’t employ A/B tests as a primary method to evaluate CRO work.
- You are a professional statistician or experimental design specialist.

## Skills you should have before taking this course:

- Some experience in conversion rate optimization.
- Basic understanding of how A/B testing works.
- Some experience with an A/B testing software.

## About

**Georgi Georgiev**

Georgi is an expert internet marketer and statistician working passionately in the areas of SEO, SEM and Web Analytics since 2004. He is the founder of Analytics-Toolkit.com and owner of an online marketing agency & consulting company: Web Focus LLC and also a Google Certified Trainer in AdWords & Analytics.

His special interest lies in data-driven approaches to testing and optimization in e-commerce and internet advertising and Georgi is also the author of three papers, multiple articles on A/B testing for conversion rate optimization, as well as the book “Statistical Methods in Online A/B Testing”.

## In just 10 sessions, you’ll be able to

- Plan A/B tests by taking into account relevant error probabilities
- Correctly interpret A/B testing statistics like p-values and confidence intervals.
- Navigate the complexities of MVT, segmentation, multiple KPIs, and concurrent tests.
- Efficiently communicate A/B test statistics.

## Your full course curriculum

#### Statistics for A/B testing

Lesson 1

### Basics of Causal Inference

**Lesson Objectives:**

- Understand the difficulties involved in making sense of data in a noisy world
- Learn how experiments help us deal with some of those difficulties
- Establish the role of statistics in making business decisions
- Translate business questions into statistical hypotheses

Lesson 2

### Statistical Significance & Other Estimates

**Lesson Objectives:**

- Understand the need for the p-value and what it means
- Learn how to calculate the p-value for absolute difference in proportions
- Introduce confidence intervals as an alternative presentation of the discrepancy between observations and a model

Lesson 3

### Statistical Power & Sample Size Calculations

**Lesson Objectives:**

- Understand type II errors, the concept of statistical power of a test, and how to account for them in planning an A/B test
- Learn how to determine the required sample size for a simple A/B test
- Understand the relationship between power, significance threshold, minimum effect of interest, and sample size
- Be able to plan a fixed-sample A/B test so it achieves a target power at a specified minimum effect of interest

Lesson 4

### Multivariate Tests

**Lesson Objectives:**

- Understand what challenges are posed by A/B/n designs & the different tools for addressing them
- Understand the implications of increasing the number of variants tested
- Explore the trade-off between testing many variants and testing quickly

Lesson 5

### Running Concurrent A/B Tests

**Lesson Objectives:**

- Explore the complications introduced by running concurrent tests
- Understand why commonly-proposed approaches do not get the job done
- Learn how and if to engage in concurrent testing

Lesson 6

### Tests With Multiple Outcomes

**Lesson Objectives:**

- Understand the difference between primary and secondary outcomes
- Learn how to recognize situations which require FWER control and which do not
- Design tests with multiple primary and secondary outcomes

Lesson 7

### Non-binomial data

**Lesson Objectives:**

- Understand the three types of metrics in A/B testing
- Cut through common misconceptions about statistics based on non-binomial data
- Calculate statistical estimates for non-binomial data

Lesson 8

### Statistics for Percentage Change

**Lesson Objectives:**

- Explore the difference between absolute and relative or percentage difference
- Define a proper statistical model for percent change
- Calculate statistical estimates for working with percent change outcomes

Lesson 9

### Asking the Right Questions

**Lesson Objectives:**

- Recognize when a one-sided test is appropriate
- Understand the need for correspondence between substantive(business) and statistical hypothesis
- Learn how to translate different business questions to appropriate statistical hypotheses

Lesson 10

### Communicating Statistical Results

**Lesson Objectives:**

- Learn how to prepare stakeholders for using statistical insights
- Explore best practices for presenting statistical insights through graphs, stories, tables, etc.

## Show off your new skills: **Get a certificate **of completion

Once the course is over, pass a test to get certified in Statistics for A/B Testing.

Add it to your resume, your LinkedIn profile, or just get that well-earned raise.

**CXL** Institute is an authorized LinkedIn education provider

You can add your education, certificates, badges – everything you learned and earned at CXL Institute into the Education section of your profile.

**Get this course** and 50+ others when you sign up for CXL Institute

This course (and all others – full list here) are included with the CXL Institute subscription.

At CXL Institute, you can take online courses on growth, digital marketing, optimization, analytics, persuasion – all in a single subscription.

- World’s leading practitioners teaching you their best stuff.
- 50+ online courses on all things data-driven marketing
- New courses added every single month

## Sign up for **CXL Institute** and get access to

- 50+ courses (full list here) on all aspects of data-driven marketing (new courses added every month)
- 4 Minidegrees that give you advanced level skills
- Talk recordings by world class practitioners at CXL Live and Elite Camp

**Got a team, multiple users? **Get our Teams All-Access plan instead.

## The *C***XL** Institute guarantee:

- 7-day no questions asked money back period
- You can immediately apply learnings and improve your results.

**Need help convincing your boss?**