Models • A/B Testing

• Overview
• Statistical power analysis
• References

Overview

Statistical power analysis

• The power calculator computes the test power based on the sample size and draw an accurate power analysis chart.

Larger sample size increases the statistical power.

• The test power is the probability to reject the null assumption, \(H_0\), when it is not correct.
• Power is expressed mathematically as \(1 - \beta\).
• Researchers usually use the power of 0.8 which means the Beta level (\(\beta\)), the maximum probability of type II error, failure to reject an incorrect \(H_0\), is 0.2.
• The commonly used significance level (\(\alpha\)), the maximum probability of type I error, is 0.05.
• The Beta level (\(\beta\)) is usually four times as big as the significance level (\(\alpha\)), since rejecting a correct null assumption consider to be more severe than failing to reject incorrect null assumption.

References

• Statistical power calculators
• How to set alpha when you have underpowered experiments?
• Raise the Bar on Shared A/B Tests: Make them trustworthy