Reports include several data points and visualization representations that help you understand the lift bounds and confidence level associated with your Adobe Target activity to help you more accurately determine a winner.
This feature is available only when viewing reports in Table View. This feature is not available for activities that use Analytics as the reporting source (A4T).
The following illustration shows Lift Bounds and Confidence Level information:
The lift and confidence information in the Target reporting UI includes:
The large number and arrow reflect the expected value of the lift. This number is the midpoint of the range of the lift bounds. The expected lift arrow displays in grey until the confidence passes 95%. After this threshold, the arrow displays as red or green based on negative or positive lift, respectively.
This is the 95% confidence interval of the lift. It displays as a range below the average lift. See Example calculation below for an example of how these lift bounds are calculated.
The boxplot graph in the Target interface represents the expected value and 95% confidence interval of the success metric in question. Think of it as a graphical way to view the lift and lift bounds information.
There are a few key ways Target helps you interpret the confidence information, one of which is color. The graph displays any overlap in the confidence interval of a specific experience with the confidence interval of the control in grey, and any range of a specific experience’s confidence interval that is above or below that of the control confidence interval as green or red, respectively.
The length of the boxplot bar represents how large the confidence interval is in an easily understood way. As you collect more data in your activity, the bar shifts and changes. The confidence interval is derived from the variance and the sample size (number of visitors). The smaller the variance and the larger the sample size, the narrower your confidence interval.
The confidence of an experience or offer that is displayed is a probability (expressed as a percentage) of obtaining a result less extreme than the one that is actually observed, if the null hypothesis is true, i.e. if there is no difference in conversion rates between that experience or offer, and the control experience/offer. In terms of p-values, this confidence displayed is
1 - p-value. Put more simply, higher confidence indicates that the data is less consistent with the assumption that the the control and non-control offer/experience have equal conversion rates.
Download the Confidence Interval for Lift pdf file for more information.
The lift bounds represent the 95% confidence intervals of the lift that the specific experience or offer has over the control experience or offer. Loosely speaking, it means the actual lift has about a 95% chance of being between these bounds.
The lift bounds are calculated using the following formula:
There is some additional calculation to arrive at the input to our lift bounds:
t-value: The critical statistic for our 95% confidence level is 1.96. You can learn more about t-values here.
Lift Variance: The Standard Error of Experience N’s success metric and the Standard Error of the Control Experience’s success metric are needed to determine the lift variance, which is calculated using the following formula (illustrated in the case the success metric is conversion).
Conversion Rate / Success Metric Standard Error: Standard error is calculated in the same way for Experience N and the Control, using the following formula (illustrated in the case the success metric is conversion). You can learn more about standard error here.
The standard error for revenue success metric activities is based on the sample variance of the revenue.
Let’s consider an example activity with two experiences and the following results:
|Experience A (Control)||219, 328||2,466||1.12%|
|Experience B||218, 362||3,040||1.39%|
Based on our formulas, we can calculate the inputs we need for the lift bounds.
Standard Error for Experience A (Control)
Standard Error for Experience B
Lift Variance for Experience B
Lift Bounds for Experience B
Expected Lift for Experience B:
Therefore, the lift bounds for Experience B would be:
Expect minor variances between manual calculations using the above formulas and the numbers displayed in the report. The difference can be attributed to the fact that the page views numbers used in manual calculations are rounded-off. The lift shown in the Target report is based on the exact numbers obtained from the total engagement and the engagement count. The engagement numbers can be obtained via the performance report API.
In certain cases, Target does not display lift bounds: