Models • A/B Testing
Overview
Statistical power analysis
- The power calculator computes the test power based on the sample size and draw an accurate power analysis chart.
Larger sample size increases the statistical power.
- The test power is the probability to reject the null assumption, \(H_0\), when it is not correct.
- Power is expressed mathematically as \(1 - \beta\).
- Researchers usually use the power of 0.8 which means the Beta level (\(\beta\)), the maximum probability of type II error, failure to reject an incorrect \(H_0\), is 0.2.
- The commonly used significance level (\(\alpha\)), the maximum probability of type I error, is 0.05.
- The Beta level (\(\beta\)) is usually four times as big as the significance level (\(\alpha\)), since rejecting a correct null assumption consider to be more severe than failing to reject incorrect null assumption.
In A/B testing, a commonly used statistical test to check for statistical significance is the t-test (specifically the two-sample t-test) if the data is normally distributed. If the data does not meet normality assumptions, the Mann-Whitney U test (also known as the Wilcoxon rank-sum test) can be used as a non-parametric alternative. These tests compare the means or distributions of the two groups to determine if there is a statistically significant difference between them.
What statistical test would you use to check for statistical significance in A/B testing, and what kind of data would it apply to?
- In A/B testing, a commonly used statistical test to check for statistical significance is the t-test (specifically the two-sample t-test) if the data is normally distributed. If the data does not meet normality assumptions, the Mann-Whitney U test (also known as the Wilcoxon rank-sum test) can be used as a non-parametric alternative. You would apply these tests to continuous metrics such as the average order value, click-through rate, or conversion rate between the two groups (A and B) to determine if there is a significant difference in their means.