AB Testing for Email Deliveries and statistical significance

Description

Environment

• Campaign Classic
• Campaign
• Campaign Standard

Issues/Symptoms
Documentation around AB Testing for Email Deliveries andstatistical significance

Resolution

In order to be statistically significant an A/B test should rely on big enough A, B and remainder segments. It is generally considered that A and B segments compose of 1500 recipients each (so 3000 as a whole). The recommendation would be that:

• If your initial target is below 30.000 recipients, then A/B testing is more or less irrelevant.
• If your target is anywhere between 30.000 and 50.000 recipients then you should shoot for A and B segments accounting each for a minimum 10% of your initial target.
• If your target is above 50.000 then you can safely shoot for A and B segments accounting each for at least 5% of your initial target.