Lift and confidence - A4T FAQ

Answer

You can perform offline calculations for A4T, but it requires a step with data exports in Analytics. For more information, see Statistical calculations in A/Bn tests.

Answer

Lift is the percent difference between your control page results and a successful test variant.

Answer

The confidence level is a probability, expressed as a percentage, that is equal to 1 - p-value, where the p-value is computed from a t-test. See Statistical calculations in A/Bn tests.

Answer

Calculated metrics are not currently supported in lift and confidence functions. Analytics calculates metrics at an aggregate-level, rather than at a visitor-level. Confidence, in particular, is a visitor-level calculation.

Non-calculated (standard) events are supported in lift and confidence. They become the numerator in the lift function; the numerator cannot be a calculation itself. The denominator is the normalizing metrics (impressions, visits, or visitors). Some examples of standard events include orders, revenue, activity conversions, custom events 1-1000, and so on. Common optimization metrics, such as conversation rate (Orders/Visitor) and RPV (Revenue/Visitor) are supported in lift and confidence.

Examples of unsupported metrics or use cases include:

• Average Order Value (Revenue/Order, per Visitor). AOV is not supported because the numerator is a calculated metric. Instead, the recommendation is to consider the two influencing metrics of AOV - Revenue Per Visitors and Conversion Rate.
• Calculated metrics that are the sum of standard events. For example, you can track ten different lead forms into ten separate events, and then add them together to get total lead submissions. A recommended method to track these events is to implement a single lead submission event in Analytics and then use an eVar to collect the type of lead form. Using this method requires fewer variables and ensures that you can use the single lead submission metric in lift and confidence functions.

Answer

Adobe Analytics treats all metrics as non-binary, and therefore, computes confidence/p-values in a manner that is different to the use of binary metrics in a regular t-test. Specifically, the calculations used by A4T allow for each user to have a continuous metric outcome (not just 1 or 0 for each user), so that the variance (or relatedly, the standard deviation) for each experience must be calculated appropriately. Extreme orders are not considered. Also, the confidence calculation does not apply a Bonferroni correction for multiple offers.

Answer

Lift and confidence do not work in Ad Hoc or Report Builder, and cannot be calculated yourself for continuous variables. It is possible to calculate it manually for binary metrics.