Tests with more than one Treatment group

When the A/B test has a control group and one treatment group, you just wait for your statistical test to read 95% significance and then declare a victor and you’re done. If there are multiple treatment groups, you might wait for 95% statistical significance for any treatment against control, then declare a victor and you’re done. But you’re really not done. If you’ve read my previous post about multiple simultaneous A/B tests, you might be able to guess one problem. But there is also a second problem.

How do I handle this situation?

If you have read my previous post, you know the first problem. If we’re testing multiple features, we’re more likely to see a large difference between test groups just by chance. Since we now testing more than one treatment against control, it is more likely that one of these treatment groups are different just by chance. This means we need to correct our test procedure. One easy correction is called a Bonferroni correction. Suppose we have one control group and 4 treatment groups so we have two tests that are being performed against the control group. We change our 100% - 5% = 95% significance level to 100% - 5%/4 = 98.75% significance. This is actually a conservative correction and there are better correction, but the Bonferroni correction is quick and easy to do.

Now, there is still a second problem. Lets say that both of our two treatment groups are higher than control and one is statistically significant. We pick the statistically significant one and we’re done right? Not exactly. Yes, one treatment group is higher than the control group. But we don’t know if that same treatment group is also higher than the other treatment group. We need to perform another statistical test comparing these two treatment groups in order to correctly pick a winner. In general, we should be comparing all pairs of groups with each other. If we have k total groups (1 control group and k-1 treatment groups), we need to perform up to (k choose 2) different pairwise comparisons to find the best group. Also also means making a Bonferroni Correction with (k choose 2) different treatment groups. So back to our example of 1 control and 4 treatment groups, this means we compare (5 choose 2) = 10 different pairwise comparisons looking for a significance level of 100% - 95%/10 = 99.5% significance.

Lets look at some examples in R. Lets use just a control and two treatments, each from a different distribution. Control has the lowest average. Treatment 1 has the middle average. Treatment 2 has the highest average.

set.seed(2015)
run_experiment_once <- function(x) {
    # Create 3 data sets with 3 different means
    control <- rnorm(100, mean=1, sd=2)
    treatment1 <- rnorm(100, mean=1.4, sd=2)
    treatment2 <- rnorm(100, mean=1.6, sd=2)
    
    # Compute significance for each of the three pairs of tests
    p_value_1 <- t.test(control, treatment1)$p.value
    p_value_2 <- t.test(control, treatment2)$p.value
    p_value_3 <- t.test(treatment1, treatment2)$p.value
    
    # Return all 3 results
    c(p_value_1, p_value_2, p_value_3, mean(control), mean(treatment1), mean(treatment2)) 
}

# Repeat the experiment 10,000 times
result <- sapply(1:10000, run_experiment_once)

Lets look at each treatment compared with control and see how often we would’ve finished our A/B test. Finishing the A/B test in this case would be finding significance in either of the treatments when compared with Control. Don’t forget to apply the Bonferroni Correction

# The first row in result is the significance between control and treatment 1
result1 <- result[1,] < 0.025

# The second row in result is the significance between control and treatment 2
result2 <- result[2,] < 0.025

# Sum will add up the list of boolean expressions where TRUE=1 and FALSE=0
count.either_success <- sum(result1 | result2)

# display the percentage of success
count.either_success / 10000

## [1] 0.4959

It looks like for our experiment, at least one of the two treatments are statistically significant 49.6%% of the time. Now, lets take a look and see how often treatment 2 was significant and not treatment 1.

count.treatment1_failure.treatment2_success <- sum(result[1,] > 0.025 & result[2,] < 0.025)

count.treatment1_failure.treatment2_success / count.either_success

## [1] 0.5922565

count.treatment1_failure.treatment2_success / 10000

## [1] 0.2937

Given that we found statistical significance, we can see that 59% of the time, we found statistical significance with treatment 2 (the higher one) before treatment 1. If the approach was to declare victory at this point, you would be correct. Overall, out of 10,000 experiments, we would’ve correctly choosen treatment 2 29% of the time.

Now, let’s take a look how often treatment 1 was significant and not treatment 2

count.treatment1_success.treatment2_failure <- sum(result[1,] < 0.025 & result[2,] > 0.025)

count.treatment1_success.treatment2_failure / count.either_success

## [1] 0.1103045

count.treatment1_success.treatment2_failure / 10000

## [1] 0.0547

Now, given that we found statistical significance, 11% of the time, we would’ve incorrectly declared treatment 1 to be the winner when the data we generated is actually lower. Overall, this would’ve been 5.5% of the time.

For our experiment, it wouldn’t be useful to check if both treatments are significant or not. This wouldn’t happen if we followed our strategy of picking the first treatment that shows significance.

Lets try the other approach. We can take a look at the highest group and see how often it was significantly higher than the other two groups. Don’t forget the Bonferroni Correction

# the fourth row is the average of control
# the fifth row is the average of treatment 1
# the sixth row is the average of treatment 2

count.control_highest <- sum(result[4,] > result[5,] & result[4,] > result[6,])
count.control_highest / 10000

## [1] 0.0065

count.treatment1_highest <- sum(result[5,] > result[4,] & result[5,] > result[6,])
count.treatment1_highest / 10000

## [1] 0.2415

count.treatment2_highest <- sum(result[6,] > result[4,] & result[6,] > result[5,])
count.treatment2_highest / 10000

## [1] 0.752

We see that control is highest 0.65% of the time. Treatment 1 is highest 24% of the time and Treatment 2 is highest 75% of the time. Now, given the group is highest, let’s see if it is also statistically significant against both other groups

# the first row is p-value for control vs treatment 1
# the second row is p-value for control vs treatment 2
# the third row is p-value for treatment 1 vs treatment 2

count.control_significance <- sum(result[4,] > result[5,] & result[4,] > result[6,] & result[1,] < 0.025 & result[2,] < 0.025) 
# given control is highest, what percent is it significant?
count.control_significance / count.control_highest

## [1] 0

# what percent is control highest and significant?
count.control_significance / 10000

## [1] 0

We can see if control is the highest, we never have statistical significance, which is correct since we generated it with the lowest mean.

count.treatment1_significance <- sum(result[5,] > result[4,] & result[5,] > result[6,] & result[1,] < 0.025 & result[3,] < 0.025) 
# given treatment 1 is highest, what percent is it significant?
count.treatment1_significance / count.treatment1_highest

## [1] 0.004968944

# what percent is treatment 1 highest and significant?
count.treatment1_significance / 10000

## [1] 0.0012

Given treatment 1 is highest, it was statistically significant only 0.5% of the time. Overall, it was highest and statistically significant only 0.12% of the time which is much better than the other approach.

count.treatment2_significance <- sum(result[6,] > result[4,] & result[6,] > result[5,] & result[2,] < 0.025 & result[3,] < 0.025)  
# given treatment 2 is highest, what percent is it significant?
count.treatment2_significance / count.treatment2_highest

## [1] 0.06263298

# what percent is treatment 2 highest and significant?
count.treatment2_significance / 10000

## [1] 0.0471

Given treatment 2 is highest, we find statistical significance 6% of the time. Overall, it was highest and statistically significant 4.7% of the time, much less frequently than the other approach. This happens since we’re testing significance against both control and treatment 1, instead of just control.

There is something better?

Since we’re still looking at significance between all pairs of tests, we’re going to need more time and data. But this is the proper thing to do. If desired, you can remove the test groups that are statistically significant however.

There is an additional approach that works. If we only look at pairwise comparisons between groups, we are ignoring all other groups and not using all of our information. We can instead rank all of our treatments and group them into a high and low group, each time choosing a different point to break up the treatments. This may give more success than looking at only two groups  at a time.

Finally, there are a various tests that test all the groups at once and tell you if all groups have all the same average. If one group has a different average, the test will fail and reject the null hypothesis. ANOVA is one type of test. You can read more about it below.

http://www.physics.csbsju.edu/stats/anova.html

https://explorable.com/anova

http://en.wikipedia.org/wiki/Analysis_of_variance

Conclusion

Now you know how to properly control your level of significance and also find the correct winner when there are more than one treatment group in your A/B test. As you can see, it will take longer to get significance against all groups instead of just control, but this is the correct thing to do.