Statistical Power: What It Is and How To Calculate It in A/B Testing
Statistical significance is only part of deciding whether an A/B test is trustworthy. A test can fail to detect a real improvement simply because it does not have enough statistical power.
Statistical power is the probability that a test will detect an effect of a specified size when that effect genuinely exists. It is usually expressed as 1 – β, where β is the probability of a Type II error, or false negative.
For A/B testing, power should be considered before the test starts alongside sample size, the minimum effect you want to detect, the significance level, and the underlying conversion rate. An underpowered test increases the risk of missing an effect that actually matters.