Average Lift, Lift Bounds, and Confidence Interval

NOTE

This feature is available only when viewing reports in Table View. This feature is not available for activities that use Analytics as the reporting source (A4T).

Interpret the data

The following illustration shows Lift Bounds and Confidence Level information:

The lift and confidence information in the Target reporting UI includes:

Lift

The large number and arrow reflect the expected value of the lift. This number is the midpoint of the range of the lift bounds. The expected lift arrow displays in grey until the confidence passes 95%. After this threshold, the arrow displays as red or green based on negative or positive lift, respectively.

Lift bounds

This is the 95% confidence interval of the lift. It displays as a range below the average lift. See Example calculation below for an example of how these lift bounds are calculated.

Boxplot graph

The boxplot graph in the Target interface represents the expected value and 95% confidence interval of the success metric in question. Think of it as a graphical way to view the lift and lift bounds information.

There are a few key ways Target helps you interpret the confidence information, one of which is color. The graph displays any overlap in the confidence interval of a specific experience with the confidence interval of the control in grey, and any range of a specific experience’s confidence interval that is above or below that of the control confidence interval as green or red, respectively.

The length of the boxplot bar represents how large the confidence interval is in an easily understood way. As you collect more data in your activity, the bar shifts and changes. The confidence interval is derived from the variance and the sample size (number of visitors). The smaller the variance and the larger the sample size, the narrower your confidence interval.

Confidence

The confidence of an experience or offer that is displayed is a probability (expressed as a percentage) of obtaining a result less extreme than the one that is actually observed, if the null hypothesis is true, i.e. if there is no difference in conversion rates between that experience or offer, and the control experience/offer. In terms of p-values, this confidence displayed is `1 - p-value`. Put more simply, higher confidence indicates that the data is less consistent with the assumption that the the control and non-control offer/experience have equal conversion rates.

How are lift bounds calculated?

The lift bounds represent the 95% confidence intervals of the lift that the specific experience or offer has over the control experience or offer. Loosely speaking, it means the actual lift has about a 95% chance of being between these bounds.

The lift bounds are calculated using the following formula:

There is some additional calculation to arrive at the input to our lift bounds:

• t-value: The critical statistic for our 95% confidence level is 1.96. You can learn more about t-values here.

• Lift Variance: The Standard Error of Experience N’s success metric and the Standard Error of the Control Experience’s success metric are needed to determine the lift variance, which is calculated using the following formula (illustrated in the case the success metric is conversion).

• Conversion Rate / Success Metric Standard Error: Standard error is calculated in the same way for Experience N and the Control, using the following formula (illustrated in the case the success metric is conversion). You can learn more about standard error here.

NOTE

The standard error for revenue success metric activities is based on the sample variance of the revenue.

Example calculation

Let’s consider an example activity with two experiences and the following results:

Experience Visitors Conversions Conversion Rate
Experience A (Control) 219, 328 2,466 1.12%
Experience B 218, 362 3,040 1.39%

Based on our formulas, we can calculate the inputs we need for the lift bounds.

Standard Error for Experience A (Control)

Standard Error for Experience B

Lift Variance for Experience B

Lift Bounds for Experience B

Expected Lift for Experience B:

Therefore, the lift bounds for Experience B would be:

NOTE

Expect minor variances between manual calculations using the above formulas and the numbers displayed in the report. The difference can be attributed to the fact that the page views numbers used in manual calculations are rounded-off. The lift shown in the Target report is based on the exact numbers obtained from the total engagement and the engagement count. The engagement numbers can be obtained via the performance report API.

When are lift bounds not displayed?

In certain cases, Target does not display lift bounds:

• For any activity, when the total number of visits or visitors is fewer than 30.
• For Auto-Allocate activities, no lift bounds are displayed until one experience has attained 60% confidence.