Documentation

AB testing for email deliveries and statistical significance

Last update: Mon May 13 2024 00:00:00 GMT+0000 (Coordinated Universal Time)

Description description

Environment

  • Campaign Classic
  • Campaign
  • Campaign Standard

Issues/Symptoms

Recommendations for AB testing for email deliveries and statistical significance.

Resolution resolution

In order to be statistically significant an A/B test should rely on big enough A, B, and remainder segments. It is generally considered that A and B segments compose of 1500 recipients each (so 3000 as a whole). The recommendation would be that:

  • If your initial target is below 30.000 recipients, then A/B testing is more or less irrelevant.
  • If your target is anywhere between 30.000 and 50.000 recipients, then you should shoot for A and B segments accounting each for a minimum 10% of your initial target.
  • If your target is above 50.000 then you can safely shoot for A and B segments accounting each for at least 5% of your initial target.
recommendation-more-help
3d58f420-19b5-47a0-a122-5c9dab55ec7f