AB Testing for Email Deliveries and statistical significance
Description
Environment
- Campaign Classic
- Campaign
- Campaign Standard
Issues/Symptoms
Documentation around AB Testing for Email Deliveries andstatistical significance
Resolution
In order to be statistically significant an A/B test should rely on big enough A, B and remainder segments. It is generally considered that A and B segments compose of 1500 recipients each (so 3000 as a whole). The recommendation would be that:
- If your initial target is below 30.000 recipients, then A/B testing is more or less irrelevant.
- If your target is anywhere between 30.000 and 50.000 recipients then you should shoot for A and B segments accounting each for a minimum 10% of your initial target.
- If your target is above 50.000 then you can safely shoot for A and B segments accounting each for at least 5% of your initial target.