Email A/B testing

An A/B test is a single-variable, randomized experiment with two variants: A and B. Its purpose is to identify which variant is more effective in changing a user’s behavior.

You can A/B test emails and landing pages with Marketo Engage. The process is the same regardless of the asset.

Click each step to learn more.

Step 1: test settings

First, determine your test settings, which identify which variable you want to test, such as the subject line of an email. You will have two versions of this variable, A and B. These versions will be sent to a random sample of your audience. For example, if your random sample is 10%, 5% will be sent version A and 5% will be sent version B.
Step 2: winner criteria
Second, you’ll define the desired behavior of the recipient that determines which version will be the “winner.” In an email subject line test, this is usually the subject line that gets a user to open the email.
Step 3: schedule
Third, you schedule your test, defining when the test will be launched to the random sample of your audience and when the winning version will be provided to the remaining audience.
Step 4: finish
Last, you’ll review and approve your A/B test.

Set up an A/B email test in Marketo Engage

In this lesson, you’ll learn how to set up an A/B test in an email using the email program A/B testing editor.

The email tile with the add A/B test button highlighted.
The variations part of the A/B test with the described test settings

Test settings

For any email A/B test, there are four test types to choose from:

  • Subject Line: test two or more subject lines against each other.

  • Whole Email: a design test allowing you to test imagery or copy placement variations.

  • From Address: test variations of the From Name and Email Address of your email.

  • Date/Time: test dates and deployment times against each other.

Winner criteria

For each test, select the winner criteria from the following list:

  • Opens: used for Subject Line, From Address, and Date/Time tests.

  • Clicks/Click to Open %: used for Design tests.

  • Engagement Score: can be used for any test because it takes multiple metrics into account. Remember that the Engagement Score considers opens, clicks, program success, and opt-outs, and provides one holistic score from 0 to 100, with 50 being average.

  • Custom Conversion: can be used with any test type, allowing you to define winning criteria by a person’s specific behavior (trigger). For example, if your email encouraged people to click through to your website, fill out a form, and download a research paper, then your Custom Conversion criteria could be which version of the email generated more form fill outs.

The Winner Criterial part of the A/B test with options as described
The Declare Winner part of the A/B test showing automatic and manual options.

Declare winner

Once you’ve determined your winner criteria, choose how you want the winning version of your email to be declared: Automatic or Manual.

In an automatic test, Marketo Engage will wait for the test to be completed and then send out the winning email at the scheduled time. You will receive a notification to confirm the test is done and that the winning version has been deployed to the remainder of your audience.
Notification email from Marketo Engage confirming A/B test completion and deployment of winner to the remainder of your audience.
With a manual test, Marketo Engage will wait for the test to be completed and then send you a notification with the test results. You will need to review the results and declare a winner to be sent at a specific time.
Notification email from Marketo Engage reminding you that the A/B test is complete and you need to take action to deploy the winner to the remainder of your audience.
style
default

Can you guess which version of this email generated more definitive guide downloads?

Guess the winner

The first photo!

Testing showed the random sample for version A received 14% more downloads than version B.

This does not mean that all of your email hero images need to look like this one. The point of A/B testing is to better understand your audience. Even if your preference was not the winner, each audience is unique, and you need to perform A/B testing to determine what works best for your goals.

style
default

Demonstration: set up an email A/B test and approve the program

Now that you know more about A/B tests, you'll learn how to build an A/B test for this product announcement email. To do this:

  1. Add an A/B test to your email program

  2. Define the test settings

  3. Define the winner criteria

  4. Schedule when the test emails and the winner will be sent automatically.

The email is ready to send out, so the expert needs to approve the final configuration. Note that they scheduled the email using the A/B test, so they do not need to schedule this on the schedule tile.

https://video.tv.adobe.com/v/3471786?learn=on
style
bg-dark-gray

In the next step, you will learn more about measuring your email program’s success.

style
default
center-aligned
false