Analyze an A/B Test
Role: Data Analyst , Data Visualization Designer
Tools Used: Python, NumPy, pandas, Matplotlib, statsmodels, Jupyter notebook, Sketch
GOAL
Analyze an A/B test to help an e-commerce website company decide if they want to implement a new page, keep the old one, or run the experiment longer.
What I Did
Use Python to analyze probability, A/B test, and regression
Determine which page converts better, the old or new
Create a custom visualization to communicate findings
View Python Code
PROCESS — Part I: Probability
I calculated the actual differences in our observed sample:
Overall conversions: 11.96%
Control (old page) conversions: 12.04%
Treatment (new page) conversions: 11.88%
The difference in conversions is small, but it appears that the old page is more effective and the new is not worth allocating resources. We might benefit from more testing, but it has been run for 22 days, which seems like an adequate amount of time especially considering that we tested over 290,000 unique users.
PROCESS — Part II: A/B Test
Null Hypothesis (assume to be true by default): the new page is worse, or only as good, as the old
Alternative Hypothesis (want to prove to be true): the new page is better than the old
To estimate population conversion rates, I created simulated samples and ran the test 10,000 times. Most data points fall within the 99% confidence interval (red lines) suggesting that our null hypothesized value did generate our statistic. The probability that our observed statistic comes from this distribution is to the right of the yellow line. This is more than half of our distribution, supporting our null hypothesis that the old page converts better.
P-value = 0.9. The p-value helps us to determine the statistical significance, which is the probability of observing our statistic, or more extreme values in favor of the alternative (greater in this case), if the null is true. With a p-value of 0.9 (greater than the Type I error threshold of 0.05), we do not have statistically significant evidence that suggests the new page converts better. Therefore, we fail to reject the null (stick with the old page).
PROCESS — Part III: Regression Analysis
I performed a regression analysis using statsmodels to confirm the results achieved in the A/B Test. I used logistic regression rather than linear since we want to predict one of two possible outcomes: whether a user will convert or not depending on the page (old or new).
In regression, the p-value tests if a parameter, in this case the slope, does or does not equal zero. In other words, is the variable useful for predicting a response? Smaller p-values (< 0.05) suggest a variable (x) is statistically significant in relating to the response variable (y); there is a slope (correlation) between x and y.
Null Hypothesis: slope != 0
Alternative Hypothesis: slope = 0
P-value for x = old page: 0.00. Statistically significant; x (old page) and y (conversions) are correlated
P-value for x = new page: 0.19. Not statistically significant; x (new page) and y (conversions) are not correlated
Conclusion
I recommend keeping the old page. Probability, A/B test, and regression analysis suggests that the old page converts better. I added regional data to observe which country/ state had better conversions. Compared to US overall conversions, CA is 1.04x less likely to convert and the UK is 1.01x more likely to convert. Perhaps CA has more change aversion and the UK is drawn to change, but these observed differences are quite small.
VISUALIZATION
I presented the process and conclusions visually, in a more understandable and engaging way, to the e-commerce company.