Advanced A/B Testing & Statistical Analysis

A comprehensive end-to-end A/B testing project that simulates user-level data to evaluate the impact of a product change on conversion rates. The project applies both frequentist (two-proportion z-test, confidence intervals) and Bayesian approaches to assess statistical significance. In addition, repeated simulations are used to evaluate the robustness of the results. Despite an initially significant outcome, simulation reveals that the result is not stable due to insufficient statistical power. The project also includes power analysis to estimate the required sample size for reliable decision-making, demonstrating strong understanding of experimental design, uncertainty, and real-world product analytics.

Key Features

• User-Level Data Simulation: Generated realistic A/B test data using binomial distributions.
• Frequentist A/B Testing: Implemented two-proportion z-test and hypothesis testing.
• Confidence Intervals: Quantified uncertainty in conversion rates for both variants.
• Bayesian A/B Testing: Estimated probability that variant B outperforms A using Beta distributions.
• Simulation Analysis: Repeated experiment 1000 times to evaluate statistical robustness.
• p-value Distribution: Visualized variability and instability of statistical significance.
• Power Analysis: Estimated required sample size to detect small effects reliably.
• Data Visualization: Created clear plots for conversion rates, posterior distributions, and simulations.
• Decision Analysis: Evaluated conflicting statistical signals and translated them into business insights.

Tech Stack

Screenshots