Campaign Standard: A/B Testing deliveries still pending while the winner is selected

This article describes why A/B testing is concluded before all the variants are delivered to recipients.

Description description

The A/B Test deliveries in a workflow were set up to send five deliveries and pick a winner based on selected criteria. However, the sending logs displayed a ‘Pending’ status, while the UI shows that a winner (‘A’) was picked, and the remaining audience received their emails. This discrepancy raised concerns about whether all intended recipients received their messages accurately.

Resolution resolution

The root cause here was related to the fact that the workflow has finished or was stopped before the delivery finished entirely.

The messages were sent and the processing of responses has started, but since the delivery activity duration is determined by the workflow the processing stopped mid way for all included deliveries.

It is important to let this type of workflow with A/B testing deliveries continue working until metrics are satisfactory.

This is due to the nature of A/B test deliveries as they have multiple phases in the flow.

Until all messages are processed, either acknowledged received, open, bounced, clicked, etc the delivery itself should not be stopped.

If the customer must utilize a workflow for A/B test deliveries, let the same workflows run indefinitely, ie. for a week for example.

Also it would be good to stagger the deliveries themselves and not the hourly rate of messages when A/B testing is done or create a separate workflow for each of those deliveries.