Because a multivariate test compares multiple experiences, it is important to know how much traffic is required to provide meaningful results. The Traffic Estimator uses statistics about your page and the number of experiences being tested to estimate the amount of traffic and the test duration needed to make the test successful.
The Traffic Estimator predicts the sample size needed to ensure the following:
This means that the chance of reporting a false positive if there is no real lift is 5% (100% - confidence level).
80% statistical power
This means that the test has a 80% probability of detecting a true lift of 25% or more.
25% minimum reliably detectable lift
Target computes the amount of traffic required to have an 80% chance of detecting a true lift of 25% or more.
The test uses the Bonferroni correction to correct for multiple comparisons. This method is known for being conservative, which is balanced out by enforcing a relatively large minimum reliably detectable lift.
The Traffic Estimator also provides feedback that lets you know whether you have sufficient traffic for the test you designed to succeed.
From the Experience Composer, click the Traffic icon.
The Traffic Estimator opens. You can click the Traffic icon again to hide the Traffic Estimator.
Provide the typical conversion rate, estimated visitors per day, and test duration.
The Traffic Estimator uses these statistics to determine what adjustments are needed to run a successful test.
Near the top of the Traffic Estimator, the values you entered are calculated and the results are shown.
As you change the numbers, the estimate changes. For example, if you are testing a large number of experiences and your conversion rate and impressions are too low, the Traffic Estimator shows how long the test will need to run to be successful. Or, if your traffic is low, the Traffic Estimator might suggest a lower number of experiences so you can run the test the desired number of days.
If you do not have sufficient traffic, you can do one or both of the following:
Adjust the numbers until the Traffic Estimator says you have sufficient traffic, then design your test accordingly.