AB Testing for Email Deliveries and statistical significance

Description

documentation around AB Testing for Email Deliveries andstatistical significance

Resolution

In order to be statistically significant an A/B test should rely on big enough A, B and remainder segments. It is generally considered that A and B segments should be at the very least each composed of 1500 recipients (so 3000 as a whole). The recommendation would be that:

  • If your initial target is below 30.000 recipients, then A/B testing is more or less irrelevant.
  • If your target is anywhere between 30.000 and 50.000 recipients then you should shoot for A and B segments accounting each for a minimum 10% of your initial target.
  • If your target is above 50.000 then you can safely shoot for A and B segments accounting each for at least 5% of your initial target.

On this page