When to categorize or assign users in A/B testing?
AB-test categorization. Part 1 - When categorization happens
When it’s mentioned that “A great way to categorize users is during the signup process”, is this specifically for testing onboarding flow changes or all changes?
The lesson also cautions against “categorizing users when they visit this page or screen” that’s subject to AB test changes. But why? Wouldn’t you want to assign users to their A/B variant at the point of change as not all users will go through the funnel and you risk requiring a larger sample size? The concern I’m flagging is also outlined here: https://medium.com/@foundinblank/why-it-matters-where-you-randomize-users-in-a-b-experiments-5570c7585944.
REPLIES
Hey Alex 
That’s an amazing question! Thank you for sharing the Medium article as well — a great read and I totally agree that the closer we’re to the changed page/screen, the smaller sample size we need.
I guess in the end, it depends on the experiment. Categorizing users on signup means that we’re getting similar cohorts — people who just found out about our app, have the same amount of information and similar intent (exlore the app, etc). If we’re AB-testing pricing page and checkout experience, I’d categorize on signup.
The worst case is that when we categorize on a pricing page (like with the described “Buy button” experiment) and have a sale at the same time, 90% of users who purchase with a heavy discount won’t look at the size of the “Buy button” and we’ll dilute the signal from new signups.
When testing a review page, for example, I’d definitely categorize a narrower cohort of users.
Does it help @Alex?