A/B Split Testing - Validity

On 15:34:13 27 July 2006 Aphillips wrote:

 

In a practical world of revenue targets, it can be difficult to always slow down long enough to stop and test and then carefully analyze your results. Understanding the validity of your data can help you to quickly make decisions and truly understand what a test is telling you.

 

 

Simply put, if you have a larger variance between two results, then you will need a smaller sample size to achieve a strong degree of confidence.

Imagine these are the results of a ficticious landing page optimization test:

 

Treatment

Unique Visits

Leads

Conversion

Landing Page A

4,203

32

0.76%

Landing Page B

3,454

534

15.46%

 

In this particular example, the difference between the number of leads is significant. Using our intuition, we can see that Landing Page B outperformed Landing Page A. However the sample size for Landing Page A Leads is still relatively small, so there is a high amount of room for error caused from sampling. There are obviously very complex algorithms for calculating the statistical relevance of a given data sample.

For a free tool for calculating validity:

Go to www.marketingexperiments.com/validity.html