From A/B Testing to Bayesian Decision-Making: A Practical Guide to Smarter Experimentation
This 3-session webinar series by Dr. Ananth Narayanan is designed to provide a comprehensive journey through key concepts in statistical reasoning and A/B testing. Each session stands alone, allowing participants to attend any talk independently, but together they form a connected narrative. We begin with the fundamentals of A/B hypothesis testing, then explore the intuitive principles of Bayesian statistics, and conclude with real-world applications of Bayesian algorithms in A/B testing. By attending the full series, participants will gain both theoretical insights and practical tools to elevate their data-driven decision-making and experimentation strategies.
First talk: Wednesday, 4/30/2025, 12:00 PM to 1:00 PM EST
A/B Hypothesis Testing: Avoiding Common Pitfalls for Reliable Decision-Making
A/B testing is a powerful statistical tool used to make data-driven decisions, but improper experimental design can lead to misleading conclusions. This introductory session requires no prior knowledge and will guide participants through the fundamental concepts of A/B testing, ensuring experiments are set up for success.
We will cover key elements of A/B testing, including experimental design, understanding p-values, types of errors (Type I & Type II), test power, and ways to improve the odds of detecting real effects. A major focus will be on demystifying statistical jargon and helping participants recognize and avoid common pitfalls such as peeking, misinterpreting p-values, and running underpowered tests.
Additionally, we will explore the advantages and limitations of A/B testing, providing a balanced perspective on when and how to apply this technique effectively. By the end of the session, attendees will have a strong foundational understanding of A/B hypothesis testing and practical insights to apply it correctly in their own work.
https://asq.webex.com/weblink/register/ra7aff0000da74f7e39a3d0f30e5983b4
Second talk: Thursday, 5/01/2025, 12:00 PM to 1:00 PM EST
Abstract: An Introduction to Bayesian Statistics: Developing an Intuitive Understanding
Bayesian statistics provides a powerful framework for reasoning under uncertainty, yet it is often perceived as complex and unintuitive. In this introductory session, we will break down Bayesian thinking using real-world examples that highlight how we naturally update our beliefs with new information—whether it’s medical diagnosis, spam filtering, or weather forecasting.
We will explore the key differences between Bayesian and Frequentist approaches, clarifying how they interpret probability and make decisions. Participants will gain a foundational understanding of marginal, joint, and conditional probabilities, essential building blocks for Bayesian reasoning. Finally, we will demystify Bayes’ Theorem, illustrating how it allows us to update probabilities as new data becomes available.
By the end of this session, attendees will have an intuitive grasp of Bayesian concepts and a solid starting point for applying them in real-world scenarios. No prior statistical background is required—just curiosity and a willingness to think probabilistically!
https://asq.webex.com/weblink/register/r1f3d8766603a084107bf1b640b1e7048
Third talk: Friday, 5/02/2025, 12:00 PM to 1:00 PM EST
Bayesian Algorithms in A/B Testing: When and How to Use Them – A Real-World Engineering Case Study
Traditional A/B testing methods can be slow and inefficient, especially in dynamic environments where rapid decision-making is crucial. This session explores Bayesian approaches to A/B testing, starting with the explore-exploit tradeoff and the multi-armed bandit problem, which form the foundation of these adaptive techniques.
We will introduce three key Bayesian algorithms—Epsilon-Greedy, Upper Confidence Bound (UCB1), and Thompson Sampling—and illustrate how they work using hypothesis testing as a case study. Through this real-world engineering example, attendees will gain an intuitive understanding of how these algorithms dynamically allocate traffic to maximize learning and performance.
Additionally, we will compare Bayesian A/B testing with traditional A/B testing, highlighting the advantages, limitations, and practical challenges of each approach. By the end of the session, participants will have a clear framework for when and when not to use Bayesian A/B testing techniques in real-world applications.
This session is ideal for engineers, data scientists, and decision-makers looking to enhance their experimentation strategies with Bayesian methods. No prior Bayesian knowledge is required!
https://asq.webex.com/weblink/register/r5bed85303f1f25c3de438f036c55a76b
About the speaker:
Ananthakrishnan (Ananth) Narayanan received his PhD in Mechanical Engineering from Auburn University. His interests include Reliability Engineering, Physics of Failure, Statistical and Probabilistic Modeling, and the application of Machine Learning and AI in Reliability Engineering. He is a subject matter expert in semiconductor/hardware reliability engineering and is currently the Principal for Worldwide Hardware Reliability at Lenovo, where he leads all technical aspects of hardware reliability for the server division. Ananth is a speaker, educator, and reliability evangelist. He has held leadership roles in various top-tier journals and conferences in the field of quality and reliability engineering. Ananth lives in North Carolina with his wife and two boys.
