AB-tests analysis Looking at AB-test performance in time

211. Looking at AB-test performance in time

We come close to the holy grail question of AB-testing. 🏆

The question is this: How confident are we in the uplift reported by our analysis? 🤔

Indeed, we have a coin and we need to determine if it’s a fair coin or not (a fair coin has 50/50 chance of getting heads or tails). After throwing the coin 3 times we have 2 tails and 1 head. Our analysis shows us a 66.67% rate of tails which is a HUGE indicator that the coin isn’t fair. We had only 3 observations, can we trust the result of this analysis?

When analyzing AB-testing results we could have...

“well worth the money”
Sign up and check out 37 free lessons and exercises.

Anatoli Makarevich, author of SQL Habit About SQL Habit

Hi, it’s Anatoli, the author of SQL Habit. 👋

SQL Habit is a course (or, as some of the students say, “business simulator”). It’s based on a story of a fictional startup called Bindle. You’ll play a role of their Data Analyst 📊 and solve real-life challenges from Business, Marketing, and Product Management.

SQL Habit course is made of bite-sized lessons (you’re looking at one atm) and exercises. They always have a real-life setting and detailed explanations. You can immediately apply everything you’ve learned at work. 🚀

“well worth the money”

Fluent in SQL in a month

Master Data Analysis with SQL with real life examples from Product Management, Marketing, Finance and more.
-- Type your query here, for example this one -- lists all records from users table: SELECT * FROM users
Loading chart... ⏳
Previous exercise: Calculating uplift for AB-test variations
Next lesson: Statistical significance, power and further reading on AB-testing