Learn about using A/B testing to improve email performance

Communicating through email is a relatively easy way to engage with leads and customers. What’s not always easy is grabbing their attention and interest, getting them to recognize and trust you and getting them to open your emails. Once they’ve opened them, it’s difficult to identify the reason they found your content engaging enough to take advantage of a special offer, sign up for an event, or download content you want to share with them or why they did not. However, if you do it right you can use A/B testing to at least determine what worked. Even if you aren’t able to tell why.

Email A/B testing is a method that helps you test a hypothesis about the behavior of your email audience. In an email A/B test, a test segment of your overall audience is randomly split into two or more equally sized test groups. Users in these test groups are then presented with different emails. One group is presented with the control and the others are presented with variations. The variations are the elements you want to test. In Marketo, you can test different focus areas of your email including the subject line, graphics, copy or the call to action. You can also test variables such as when the email is sent and who it is sent from.

After the test, you compare the results of the variations against the performance of the control. And send the winning email to everyone in your target audience, who wasn’t included in the test. If one of the variations is more successful than the control, use elements of that in a future email as the new control and continue to test it against new variations. If the control was the winner, keep using it as a control, but continue to test variations.

A/B testing helps you make data driven decisions to improve the effectiveness of your emails. Not sure of the best time to send an email to your audience? Running a series of A/B tests at different times of the day and days of the week will help you provide the answer. Have lots of ideas about subject lines but need to know which is the best? Use A/B testing to let your audience’s response help you decide. You can test almost anything that goes into an email to determine what works best. Using the results of A/B test, you’ll be able to make improvements to your emails, which will drive improvement in conversions and ultimately sales. In fact, A/B testing is probably the most cost effective way to help you improve email performance.

Once you decide to start using A/B testing, you’ll need to create a control email called, a champion, which you’ll test against one or more challengers. These contain the variations you’ll test. You can run four types of email A/B test in Marketo. Subject line, from address, date/time, and whole email.

The subject line test type is used to determine which subject line is most effective in getting people to open the email. The from address test type helps establish which from address is recognized and trusted by your audience.

The day/time test type is used to determine what time of day or day of week is best to send emails to your database. You can use the whole email test type to test elements inside the email, such as copy, calls to action, images and other items.

Once you’ve created your test variations, you must set the audience sample size to serve as the test group, sample size matters. If your test audience is too small, the results of your A/B test may be statistically misleading. Next, we choose the winner criteria settings. There are several metrics that you can use in Marketo to determine the winner of your A/B test. Open percentage, which shows the percentage of each audience that opened the emails. Click percentage, which shows the percentage of each audience that click the link inside the email, such as a call to action. Click to open percentage, which shows the percentage of your audience who opened the email and clicked a link inside of it. Engagement score, which takes into account, opens, clicks and unsubscribes. To give people a chance to engage with your content, the score is calculated 72 hours after the email is sent. Custom conversion, which you can use to determine success using any variety of different Marketo triggers. You have the ability to determine whether you want Marketo to automatically declare a winner based on your chosen metric, or if you would rather manually declare and schedule the winner to go out to the remainder of your audience. The last step is to set the date and duration of your test, then finalize it to be sent.

The email send program is unique in that it comes equipped with a built in email reporting dashboard to make reviewing performance easy. When you send an A/B test, this dashboard will provide a visual comparison of the delivery, click, click to open and unsubscribe rates for each variation. This information can be used to improve email communications as well as set the baseline for future A/B tests.

Here are a few best practices to keep in mind when you run an A/B email test. It’s best to only test one variable at a time.

For example, if you test a call to action and a hero image, in the same test, you won’t know which one is affecting your test metric or how each variable would affect a separate test of its own. Test as big a sample as possible. Larger test pools are statistically better at providing more accurate results. Just don’t make your sample so large that everyone in your database is involved in the test. After all you want to send the winning email out to the majority of your audience. Run your test at the same time and the same day of the week. Testing on different days or different times adds a new variable to the test and can skew the results. If you want to test time of day or day of week test only that. Allow enough time for your A/B test to perform before declaring a winner. Recipients may engage with your email several hours to several days after it was delivered. As a rule, you should allow at minimum 24 hours to pass before declaring a winner. Use A/B testing on all your emails. If you are running email nurture campaigns, you can A/B test all emails in your streams using a variety of test types. Track the results of your A/B tests and develop a set of email best practices for each of your audience segments. One segment may respond to emails with lots of images, where another segment responds better to emails that contain only text. Some audience members may trust one from email over another, and some may be more prone to open emails on Tuesdays at 10:00 AM, while others more reliably open them and click through later in the day. Keep track of all these preferences and use them to your advantage. Now that you’re familiar with what email A/B testing is and how it works, let’s go ahead and set it up in Marketo. -