A/B Split Testing - Validity

a very simple way to enhance the reliability of A/B testing is to switch to A/B/A testing.

as a simplistic example: you send an email to 999 people. instead of splitting that group into 2, you split it into 3 groups of 333 (group A1, group B, group A2). you use one subject line for both of the A groups, and a different subject line for the B group. This might result in something like the following:

group A1: 50 emails opened

group B: 150 emails opened 

at this stage, it looks like the 'B' subject line has totally outperformed the 'A' subject line. But can you be sure that the results are purely caused by the subject line? Perhaps the email took 3 hours to send, and the 'B' group all got their emails a little later than the 'A1' group. Perhaps the email broadcast engine plucked the 'A1' group from the start of your database (ie. they're older email addresses). Perhaps the 'B' group just coincidentally contained a large group of your best clients. The results of 'A2' can help to answer that question:

• say group 'A2' resulted in 200 emails opened, you could summarise that there was little difference between results caused by the 2 subject lines, & that your test wasn't totally reliable.
• say group 'A2' resulted in 100 opened emails, you could summarise that subject B had outperformed subject A, but that the test wasn't totally reliable.
• or - best case scenario, say group 'A2' resulted in 50 emails opened, you can be fairly happy that the test was reliable, and that subject B had drastically outperformed subject A.

The system is by no means bulletproof, but it's a really simple add-on to A/B testing that can tell you a lot about the reliability of your results, and can quickly highlight any large problems with the way you're carrying out tests.