Exploratory Data Analysis: Outliers

Outliers can easily cause problems with your A/B test. You may have seen strange anomalies with your data metrics, a particular metric being too high or too low compared to the others. You may have seen your statistical test first start significant and then become not significant. These problems may be coming from outliers in your data.

Let look at an example in R. We will create two data sets, the first data set less than the other one.

set.seed(2015)
# Create a list of 100 random draws from a normal distribution 
# with mean 1 and standard deviation 2
data1 <- rnorm(100, mean=1, sd=2)
# Create a second list of 100 random draws from a normal distribution 
# with mean 2 and standard deviation 2
data2 <- rnorm(100, mean=2, sd=2)
# Perform a t-test on these two data sets and get the p-value
t.test(data1, data2)$p.value

## [1] 0.0005304826

We can see this t-test will give a p-value of 0.0005 which is a significance level of 99.95%. Now lets add a single outlier into the first data set.

# append 1000 to the first data set only
data1 <- c(data1, 1000)
# Perform a t-test on these two data sets and get the p-value
t.test(data1, data2)$p.value

## [1] 0.369525

Now, you can see, even though we had 100 points in each data set, a single large outlier caused our data to become non-significant with a 63% significance level only

How do we fix this?

There are multiple ways to fix this problem. Student’s t-test is not robust against outliers and we can run Mann-Whitney U test instead. A deeper discussion of this approach is outside the scope of this article.

We can also detect the outliers and consider removing them. This approach needs to be taken very carefully. We should only remove data that does not come from our target population. If we see one example data point that is an outlier, it may be unlucky to see such a strange data point. However, it may also be unlucky to see only one one such data point. Therefore, outliers need to be removed only after careful examination. For A/B testing, this usually means removing data that is coming from bots and not humans. This can be difficult because not all bots and scripts report their User Agent properly.

Lets look at some diagnostic tools with R. We will create 100 points from the same distribution. Then, we will add 2 outliers to our data set.

set.seed(2015)
# Create a list of 100 random draws from a normal distribution 
# with mean 1 and standard deviation 2
data <- rnorm(100, mean=1, sd=2)

# lets add two outliers
data <- c(data, 20)
data <- c(data, 40)

# Create an image with two plots side by side
par(mfrow=c(1, 2))
hist(data)
boxplot(data, main="Box plot of data")

On the left is a histogram. We can see the outlier at 40. It is more questionable if 20 is an outlier. For the boxplot on the right, the box itself contains the 25% to 75% of the data. The thick line in the middle of the plot is the median. The “whisker” at the top and bottom of the plot are the min and max of the data except for “outliers”. A good explanation of outliers in box plots in R can be found at the bottom of this page http://msenux.redwoods.edu/math/R/boxplot.php

So now we have found a few outliers in our data. Remember, it is important to carefully consider each point before removing them, since we easily could have seen more data at that point rather than only one. One technique to try is to perform the test again but with the point removed. If the test gives the same result, then we might as well leave the data point in.

Outliers in more than one dimension

If your data contains two variables, there is another type of outlier to look for. Lets look at this plot. It has 100 points again, but with two correlated variables. Then, we add a single outlier.

set.seed(2015)
# Create a list of 100 random draws from a normal distribution 
# with mean 1 and standard deviation 2
data1 <- rnorm(100, mean=1, sd=2)

# Lets create a second correlated variable.
correlation <- 0.95
data2 <- correlation * data1 + sqrt(1-correlation) * rnorm(100, mean=1, sd=2)

#Lets add our outlier
data1 <- c(data1, -3)
data2 <- c(data2, 4)

par(mfrow=c(1, 1))
plot(data1, data2)

Here most of the data lies close to the lower-left to upper-right diagonal. We have a single point on the upper left of the plot. In any single dimension that particular point is right in the range of the data. But combined in two dimensions, it becomes an outlier.

We can find this point by computing something called Leverage. Though this is generally used to find outliers during linear regression, we can use it here to help detect some outliers.

# create a matrix with our two data sets
data_matrix <- matrix(c(data1, data2), nrow=101, ncol=2)
tail(data_matrix)

##              [,1]        [,2]
## [96,]   0.1888523  0.37660990
## [97,]  -2.3504425 -2.08643945
## [98,]   0.9110532  0.06500559
## [99,]  -1.0946785 -1.09385952
## [100,] -2.4602479 -2.40095114
## [101,] -3.0000000  4.00000000

# leverage is also knows as hat values
leverage <- hat(data_matrix)
tail(leverage)

## [1] 0.01121853 0.03700946 0.02901691 0.02292416 0.04241248 0.71791422

Above are the last 6 rows of the matrix and you can see our outlier as the last point. The corresponding leverage values are also given. You can see the very high leverage value for the last point. As a rule of thumb, leverage values that exceed twice the average leverage value should be examined more closely. However, for an A/B test, we have many observations and a wide range of leverage values. In this case, I would start examining the highest leverage points and work your way down.

Alternatives

As mentioned above, Student's t-test is not very robust to outliers. There are other tests that are more robust to outliers and are based on each observation's ranks instead of actual value. You can start looking at the Mann-Whitney U test and enter the world of non-parametric statistics

http://en.wikipedia.org/wiki/Mann%E2%80%93Whitney_U_test

http://en.wikipedia.org/wiki/Nonparametric_statistics

http://www.originlab.com/index.aspx?go=Products/Origin/Statistics/NonparametricTests

http://support.minitab.com/en-us/minitab/17/topic-library/basic-statistics-and-graphs/hypothesis-tests/nonparametrics-tests/understanding-nonparametric-tests/

http://sphweb.bumc.bu.edu/otlt/MPH-Modules/BS/BS704_Nonparametric/BS704_Nonparametric2.html

Additional Information

Here are some other links about leverage and other types of outlier detection

http://en.wikipedia.org/wiki/Leverage_(statistics)

http://en.wikipedia.org/wiki/Outlier

http://onlinestatbook.com/2/regression/influential.html

http://pages.stern.nyu.edu/~churvich/Undergrad/Handouts2/31-Reg6.pdf

Conclusion

Bots and scripts can cause problems in your A/B tests. It is important to try to detect these users in your data and remove them since they do not represent your target population.

Users or sessions?

Do you collect data at the user level or at the session level? Are treatments assigned to to each user or to each session? And is your data aggregated by user or session? The answer to both of these questions should be by user.

Why by User?

When collecting your data, it is better to assign test groups to each user instead of each session. When a user comes to the site and sees a feature, the feature may or may not affect the user during this session, but may take repeated sessions before he or she acts on it.

There is also a statistical reason. You may remember an acronym IID from your statistics class. It stands for independent and identically distributed. This refers to your sample and that it should be IID. For this article, we’re concentrating on independent samples. Independence can be described that by knowing one data point, you don’t know anything additional about any other data point. For our purposes, if your data points are all sessions, then once you know one session from a user, you have a better idea of the other sessions from that same user.

If your data isn’t independent this causes problems in your variance and error calculations. The mean of your data will stay the same, but your standard errors will be different. Having multiple observations from the same person is called clustered sampling. This requires a specific way to compute variance of your sample. Suppose the people in your data set vary quite a bit, but each observation from a specific user is exactly the same. This will cause your observed variance to be lower than the true variance. If you were to compute variance without considering the clustering, it will be underestimated.

Let’s do an example in R. Lets first create 300 data points from 300 users and compute the mean and variance. Then create 3 data data points from each of 100 users then compute the mean and variance. We will run this 1,000 times and look at the 1,000 differences. In this case, the variance between users will be the same as the variance for each user.

set.seed(2015)
library(survey)

samplesize.independent <- 300
samplesize.dependent <- 100

run_once <- function(i) {
    
    # Create 300 people each with a different mean and variance
    population.mean <- rnorm(samplesize.independent, mean=0, sd=1)
    population.var <- abs(rnorm((samplesize.independent), 
                                 mean=0, sd=1))

    # Create one data point for each user with their mean and variance
    points.independent <- mapply (function(m, v) {
                               rnorm(1, mean=m, sd=sqrt(v))
                          }, population.mean, population.var)
    points.independent <- unlist(as.list(points.independent))
    
    # create the design object, where each row is a different user
    df.independent <- data.frame(id=1:samplesize.independent,
                                 point=points.independent)
    design.independent = svydesign(id=~id, data=df.independent, 
                                   weights=~1)
    
    # compute the mean and the mean's standard error
    mean.independent <- coef(svymean(~point, design.independent))
    # mean.independent is just same as below
    # mean(df.independent$point)

    se.independent <- SE(svymean(~point, design.independent))
    # se.independent is the same as this calculation below
    #sd(df.independent$point)/sqrt(nrow(df.independent))

    # Create 100 people, each with a different mean and variance, 
    # but with same parameters as above
    population.mean <- rnorm(samplesize.dependent, mean=0, sd=1)
    population.var <- abs(rnorm(samplesize.dependent, mean=0, sd=1))

    # Create 3 data points for each user with same parameters as above
    pointsperuser<- samplesize.independent/samplesize.dependent
    points.dependent <- mapply (function(m, v) {
        rnorm(pointsperuser, mean=m, sd=sqrt(v))
    }, population.mean, population.var)
    points.dependent <- unlist(as.list(points.dependent))

    # compute the design object, setting the id to define each user
    df.dependent <- data.frame(id=sort(rep(1:samplesize.dependent, 
                          pointsperuser)), point=points.dependent)
    design.dependent = svydesign(id=~id, data=df.dependent, 
                                 weights=~1)
    
    # compute the mean and the mean's standard error
    mean.dependent <- coef(svymean(~point, design.dependent))
    # mean.independent is the same as below
    #mean(df.dependent$point)

    se.dependent <- SE(svymean(~point, design.dependent))
    # se.dependent is no longer the same as below
    se.dependent.wrong <- sd(df.dependent$point) /                
                                 sqrt(nrow(df.dependent))

    c(mean.independent, se.independent, mean.dependent, 
           se.dependent, se.dependent.wrong)
}

result <- sapply(1:1000, run_once)

# Lets look at the percentiles of the difference in means 
quantile(result[3,]-result[1,], c(0.025, 0.25, 0.50, 0.75, 0.975))

##         2.5%          25%          50%          75%        97.5% 
## -0.254349807 -0.091006840  0.007049011  0.100500093  0.278679862

# Lets look at the percentiles of the difference in variance
quantile(result[4,]-result[2,], c(0.025, 0.25, 0.50, 0.75, 0.975))

##       2.5%        25%        50%        75%      97.5% 
## 0.01619895 0.02914339 0.03480557 0.04017795 0.05136426

# Lets look at the percentiles of the difference in incorrectly 
# computed variance
quantile(result[4,]-result[5,], c(0.025, 0.25, 0.50, 0.75, 0.975))

##       2.5%        25%        50%        75%      97.5% 
## 0.02582071 0.03208966 0.03520337 0.03811021 0.04305041

We can see that the difference in mean is around zero. The 95% confidence interval is [-0.25, 0.28]. This is just as expected.

We also see that the 95% confidence interval for the difference in standard error of the mean is [0.016, 0.051], meaning the dependent points have a higher variance than the independent points. We can also see if that we compute the standard error without considering the clustering, this will also lead to a standard error that is too small, with a confidence interval of the difference from [0.26, 0.43]. This will lead to declaring significance when we don’t really have it.

Why is this happening? Here is a plot with just twelve points.

set.seed(2015)
# Create 100 users' mean and variance
mean <- rnorm(12, mean=0, sd=10)
var <- rexp(12, rate=1)

# Lets create three data points for each user, using the mean and variance from above
points <- mapply (function(x, y) {rnorm(6, mean=x, sd=sqrt(y))}, mean, var)

par(mfrow=c(2, 1))
stripchart(points[1,], xlim=c(-20, 10), main="12 independent points")

stripchart(c(points[,1], points[,6]), xlim=c(-20, 10), main="6 points from 2 users")

You can see in the bottom plot, the points are clustered and more spread out. This is what gives us higher variance.

That said, there are many assumptions that are required for a perfect statistical test. However, a lot research has been done trying to find out how much we can deviate from these assumptions. It would be impossible for your data points to be completely independent, so some departure from this assumption is expected. But staying as close as possible to the independence assumption would be the safest thing to do and shouldn’t need to be demonstrated. It should be necessary to demonstrate that this assumption can be relaxed.

Additional Information

Here are some links that talk about sampling and/or cluster sampling. The last three links contains formulas and derivations.

http://en.wikipedia.org/wiki/Sampling_(statistics)

http://en.wikipedia.org/wiki/Cluster_sampling

http://stattrek.com/survey-research/cluster-sampling.aspx

http://www.stat.purdue.edu/~jennings/stat522/notes/topic5.pdf

http://ocw.jhsph.edu/courses/statmethodsforsamplesurveys/PDFs/Lecture5.pdf

http://www.ph.ucla.edu/epi/rapidsurveys/RScourse/chap5rapid_2004.pdf

Conclusion

Please keep each user to a single test group and aggregate your data by user. Otherwise, you may declare significance when there is none.

Adjusting Traffic proportions

Let’s say you are A/B testing a new feature and you’ve given that feature 1% of the traffic and 99% of traffic to control. After a week, the feature is looking promising, so you give the feature 10% of the traffic. And as you gain more statistical significance that the feature is working well, you keep giving it a larger portion of the traffic. traffic. After a month, your A/B test has statistical significance and you declare victory. Right? Not really.

What the matter?

As the test went on, the proportion of traffic was changed between control and treatment. During the test, the content of overall traffic will change. Then the control average will be weighted more towards the old traffic and the treatment average will be weighted more towards the new traffic.

Lets try an example in R. Suppose because of marketing efforts, or because of other successful tests, our traffic suddenly changed and our metric is higher during the second half of our experiment. Now lets have a treatment that actually has no effect, but we will increase our proportion in this second half from 10% to 50% of our data

set.seed(2015)
# For the first half of the experiment
# create a list of 100 random draws from the specific Normal distribution with mean=1
first_half <- rnorm(100, mean=1, sd=1)

# For the second half of the experiment
# create a second list of 100 random draws from a normal distribution with mean=2
second_half <- rnorm(100, mean=2, sd=1)

# Define control as having 90% of traffic from the first half
# and 50% of traffic from the second half
control <- c(first_half[1:90], second_half[1:50])

# Define treatment as having 10% of traffic from the first half
# and 50% of traffic from the second half
treatment <- c(first_half[91:100], second_half[51:100])

# Perform t-test
t.test(control, treatment)$p.value

## [1] 0.002112838

Here, we pretend treatment has no effect over control. We created a before and after distribution that treatment and control both used. All we did was change was the sampling proportion of before and after. As a result, even though treatment and control should show no significance, we can see that the test is showing a 0.002 p-value which is 99.8% significance.

What can we do about it?

We can adjust our statistical test to handle the situation. The adjusting the estimated mean and variance for different proportions is out of the scope of this article.

However, there is a simple way to avoid the situation entirely. If you want to start with 10% of traffic on the new treatment, only put 10% of traffic control. Then if you want to increase traffic to treatment, just add the same amount of traffic to control.

Lets continue our last example but this time we use a proportion of 10%/10% of control/treatment before the change, and then 50%/50% after the change

# Define control to be 10% from the first half
# and 50% of traffic from the second half
control <- c(first_half[1:10], second_half[1:50])

# Define treatment to be 10% from the first half
# and 50% from the second half
treatment <- c(first_half[91:100], second_half[51:100])

# Perform t-test
t.test(control, treatment)$p.value

## [1] 0.3189567

We can see the p-value is 32% which translates into into 68% significance. This non-significance is what we expected since we used a treatment with no effect.

There is one last note. If we keep both proportion of control and treatment exactly equal, this would mean that we can only ever have up to 50% of traffic on the new treatment. If you eventually want to have 80% of traffic on treatment, then start with 8% of traffic on treatment and 2% on control and increase proportionally to 80%/20% traffic. However, it is worth mentioning that the fastest way to statistical significance is with a 50/50 split of traffic.

Additional Information

If you absolutely must change your traffic proportions, then you need to adjust your mean and variance using over sampling and under sampling techniques. In this case, once the traffic proportions have changed, you have over sampled with respect to time and this should be fixed.

http://en.wikipedia.org/wiki/Oversampling_and_undersampling_in_data_analysis

http://www.data-mining-blog.com/tips-and-tutorials/overrepresentation-oversampling/

Conclusion

Becareful when shifting more traffic to your treatment group. Make sure the ratio of traffic is always the same between your control and all treatment groups.