A/B Testing
Slides
Slides from class
Google Doc
Links
- What is A/B Testing?
- Break users into randomly assigned groups
- Show each two SMALL variations (for features, not for entire sites)
- Optimization of features, not usually for huge changes
- Must have a clear Key Performance Indicator (KPI)
- Must have single measurable action
- Benefits
o See https://www.nngroup.com/articles/putting-ab-testing-in-its-place/
- Measures real user behavior, instead of relying on your flawed insights
- Can measure really small differences that can have big impact
- Amazon bigger button, 1% increase, so conversion rate goes from 2% to 2.02%. Worth it?
- That's $69 million / year
- Resolve tradeoffs when there's conflict
- Coupon vs not? Usually 25-50% more sales when NO coupon is required. Doesn't necessarily apply to all sites. If you make more money with the coupon, use that
o Cheap
- Downsides
- Basically no insight into why
- Not generalizable (a bigger button may work on your site, but that's not evidence it will work on all sites)
- Statistical significance
- How many people to you need to find a significant difference at a given % difference?
- In practice
- Make sure people remain in the same group over the period of the test (cookies)
- Start with a small set and work up to 50/50
- Have an abort procedure if there are really bad implications from the change
Exercise: Design an A/B Test for Twitter
- What do you want to test?
- What will you measure
- How will you split it
- Are there ethical concerns? Policy concerns?
- How many people do you expect will participate / what percentage? How many do you need to reach statistical significance? (use table)