Use path experimentation experimentation
Experimentation allows you to test different paths based on a random split to determine which performs best based on predefined success metrics.
To set up path experimentation in a journey, follow the steps below.
Let’s say you want to compare three paths:
- one path with one email;
- a second path with a Wait node of two days and an email;
- a third path with an email and then an SMS message.
-
From the Orchestration section, drag and drop the Optimize activity into the journey canvas.
-
Add an optional label, which can be useful to identify the activity in reporting and test mode logs.
-
Select Experiment from the Method drop-down list.
-
Click Create experiment.
-
Select the Success metric you want to set for your experiment. Learn more on the available metrics and how to configure the list in this section.
-
Select the Experiment type for your path experiment:
-
A/B experiment — Define the traffic split between treatments at the start of the test. Performance is evaluated based on your chosen primary metric; reporting shows the observed lift between treatments.
-
Multi-armed bandit — Traffic split between treatments is handled automatically. Every 7 days, performance on the primary metric is reviewed, and weights are adjusted accordingly. Reporting continues to show lift, as for A/B tests.
➡️ Learn more about the difference between A/B and Multi-armed bandit experiments
-
-
You can choose to add a Holdout group to your delivery. This group will not enter any path from this experiment.
note note NOTE Switching on the toggle bar will automatically take 10% of your population. You can adjust this percentage if needed. -
You can allocate a precise percentage to each Treatment, or simply switch on the Distribute evenly toggle bar.
-
Enable the auto-scale experiment to automatically roll out the winning variation of your experiment. Learn more on how to scale the winner
-
Click Create.
-
Define the elements you want for each branch resulting from the Experiment, for example:
-
Drag and drop an Email activity onto the first branch (Treatment A).
-
Drag and drop a Wait activity of two days onto the first branch, followed by an Email activity (Treatment B).
-
Drag and drop an Email activity onto the third branch, followed by an SMS activity (Treatment C).
-
-
Optionally, use the Add an alternative path in case of a timeout or an error to define a fallback action. Learn more
-
Publish your journey.
Once the journey is live, users are randomly assigned to go down different paths. Journey Optimizer tracks which path performs best and provides actionable insights.
Follow the success of your journey with the Journey Path Experiment report. Learn more
Experiment use cases uc-experiment
The following examples show how to use the Optimize activity with the Experiment method to determine which path works best overall.
Test whether sending the first message by email versus SMS drives higher conversions.
➡️ Use the conversion rate as the success metric (for example: purchases, sign-ups).
Run an experiment to check if sending one email versus three emails over a week results in more purchases.
➡️ Use purchases or the unsubscribe rate as the success metric.
Compare a 24-hour wait versus a 72-hour wait before a follow-up to determine which timing maximizes engagement.
➡️ Use the click-through rate or revenue as the success metric.
Scale the winner scale-winner
Scale the Winner enables you to automatically or manually roll out the winning variation of an experiment to your full audience. This feature ensures that, once a winner is determined, you can amplify its reach and effectiveness without constantly monitoring the experiment.
You can choose between two modes:
-
Auto-scaling: Configure auto-scaling settings when creating your experiment by choosing the timing and conditions for scaling the winning treatment or a fallback option if no winner emerges.
-
Manual Scaling: Manually review experiment results and initiate the rollout of the winning treatment, maintaining full control over timing and decisions.
Auto-scaling autoscaling
Auto-scaling lets you set predefined rules for when to roll out the winning treatment or a fallback—based on the experiment’s results.
Note that once auto-scaling has occurred, manual scaling is no longer available.
To enable auto-scale in your experiments:
-
Set up your journey and configure your experiment as needed. Learn more
-
Enable the auto-scale option when setting up your experiment.
-
Select when the winner should be scaled:
- As soon as winner is found.
- After experiment is live for the selected time.
The auto-scale time must be scheduled before the experiment’s end date. If it is set for a time after the end date, a validation warning will appear, and the journey will not be published.
-
Choose the fallback behavior if no winner is found by scale time:
- Continue experiment till its ends as scheduled.
- Scale the alternative treatment after a specified time.
Once all parameters are met, your winning or alternative treatment is sent to your audience.
Manual scaling manual-scaling
Manual scaling gives you the ability to review experiment results and decide when to roll out the winning treatment on your own schedule.
Note that if you manually scale the winner before the scheduled auto-scale time, the auto-scale is canceled.
To manually scale the winner of your experiments:
-
Set up your journey and configure your experiment as needed. Learn more
-
Let the experiment run until a winner is identified or statistical significance is achieved.
-
Open your journey and select the Optimize activity that contains the path experiment.
Review the results in the Path experiment view to identify the top-performing treatment.
-
Click Scale treatment to push the winning treatment to the rest of your audience.
-
Select the treatment you want to scale from the drop-down menu and click Scale.
Note that scaling the treatment may take up to one hour. You will receive a notification once the manual scaling process is finished.