This topic contains answers to questions that are frequently asked about lift and confidence when using Adobe Analytics as the reporting source for Adobe Target (A4T).
You can perform offline calculations for A4T, but it requires a step with data exports in Analytics. For more information, see Statistical calculations in A/Bn tests.
Lift is the percent difference between your control page results and a successful test variant.
The confidence level is a probability, expressed as a percentage, that is equal to
1 - p-value, where the
p-value is computed from a t-test. See Statistical calculations in A/Bn tests.
Calculated metrics are not currently supported in lift and confidence functions. Analytics calculates metrics at an aggregate-level, rather than at a visitor-level. Confidence, in particular, is a visitor-level calculation.
Non-calculated (standard) events are supported in lift and confidence. They become the numerator in the lift function; the numerator cannot be a calculation itself. The denominator is the normalizing metrics (impressions, visits, or visitors). Some examples of standard events include orders, revenue, activity conversions, custom events 1-1000, and so on. Common optimization metrics, such as conversation rate (Orders/Visitor) and RPV (Revenue/Visitor) are supported in lift and confidence.
Examples of unsupported metrics or use cases include:
Adobe Analytics treats all metrics as non-binary, and therefore, computes confidence/p-values in a manner that is different to the use of binary metrics in a regular t-test. Specifically, the calculations used by A4T allow for each user to have a continuous metric outcome (not just 1 or 0 for each user), so that the variance (or relatedly, the standard deviation) for each experience must be calculated appropriately. Extreme orders are not considered. Also, the confidence calculation does not apply a Bonferroni correction for multiple offers.
Lift and confidence do not work in Ad Hoc or Report Builder, and cannot be calculated yourself for continuous variables. It is possible to calculate it manually for binary metrics.