AB Testing for Email Deliveries and statistical significance

Last update: 2022-11-08

Description

Environment

  • Campaign Classic
  • Campaign
  • Campaign Standard

Issues/Symptoms
Documentation around AB Testing for Email Deliveries andstatistical significance

Resolution

In order to be statistically significant an A/B test should rely on big enough A, B and remainder segments. It is generally considered that A and B segments compose of 1500 recipients each (so 3000 as a whole). The recommendation would be that:

  • If your initial target is below 30.000 recipients, then A/B testing is more or less irrelevant.
  • If your target is anywhere between 30.000 and 50.000 recipients then you should shoot for A and B segments accounting each for a minimum 10% of your initial target.
  • If your target is above 50.000 then you can safely shoot for A and B segments accounting each for at least 5% of your initial target.

On this page