Chapter 8: AB-tests analysis
This chapter on AB-test analysis gives you a head start into running and analyzing AB-tests. It goes way beyond “which button has a higher CTR” approach and gives you tools to holistically analyze AB-tests.
What is AB-testing?
AB-testing is a tool to learn about our users behavior, test new designs, features, it’s many things.
In short, during an AB-test users are split into cohorts and each cohort sees their unique AB-test variation – a completely different UX. Later, we look at the performance of each variation and trying to find a winner – a variation that later will stay in the app for all users.
AB-test mindset
AB-test analysis requires a certain mindset. Instead of blindly comparing a CTR or a single metric to find a winning variation, I highly recommend to learn why it is the case.
In other words, I recommend planning every AB-test as a proper behaviorial experiment. Start by forming a hypothesis in the beginning and then use Data Analysis to either confirm or deny this hypothesis.
This is where Funnel Analysis comes handy – we can compare an entire AARRR funnel for every variation and see its effects. This is exactly what you’ll be doing in this chapter, let’s go!
-
1. Intro to AB-testingStart
-
2. AB-testing vocabularyStart
-
3. AB-test categorization. Part 1Start
-
4. AB-test categorization. Part 2Start
-
5. Building AB-test cohortsStart
-
6. Calculating activation rate for AB-test variationStart
-
7. Building basic AB-test funnelStart
-
8. Using multiple metrics in AB-test analysisStart
-
9. Calculating ARPU of AB-test variationStart
-
10. Unit economics in AB-test analysisStart
-
11. Purchased products of AB-test variationStart
-
12. Retention in AB-test funnelStart
-
13. Calculating uplift for AB-test variationsStart
-
14. Looking at AB-test performance in timeStart
-
15. Statistical significance, power and further reading on AB-testingStart
-
16. AB-test analysis recapStart
Lessons and exercises
![Anatoli Makarevich, author of SQL Habit]()
About SQL Habit
About SQL Habit
Hi, it’s Anatoli, the author of SQL Habit. 
SQL Habit is a course (or, as some of the students say, “business simulator”). It’s based on a story of a fictional startup called Bindle. You’ll play a role of their Data Analyst 
and solve real-life challenges from Business, Marketing, and Product Management.
SQL Habit course is made of 13 chapters (you’re looking at one atm) that contain 273 bite-sized lessons and exercises. All of them have a real-life setting and detailed explanations. You can immediately apply everything you’ve learned at work. 
The 2nd part of the course is called Practice. It’s made of standalone exercises based on multiple datasets – E-commerce, Finance and Meditation app a-la Headspace or Calm. Practice exercises are harder than in the main course. They’ll get you ready for any challenge at work or an interview. 