Experimentation panel

The Experimentation panel lets analysts compare different user experiences, marketing, or messaging variations to determine which is best at driving a specific outcome. You can evaluate the lift and confidence of any A/B experiment from any experimentation platform: online, offline, from Adobe solutions like Target or Journey Optimizer, and even BYO (bring-your-own) data.

Read more about the integration between Adobe Customer Journey Analytics and Adobe Target.

Access control

The Experimentation panel is available to use by all Customer Journey Analytics users. No Admin rights or other permissions are required. However, the prerequisites require actions that only administrators can perform.

Functions in calculated metrics

Two advanced functions are available: Lift and Confidence. For more information, see Reference - advanced functions.

Prerequisites

To use the experimentation panel, ensure you follow these prerequisites:

Create connection to experiment datasets

The recommended data schema is for the experimentation data to be in an Object array that contains the experiment and variant data in two separate dimensions. Both dimensions need to be in a single object array. If you have your experimentation data in a single dimension (with experiment and variant data in a delimited string), you can use the substring setting in data views to split the dimension into two for use in the panel.

After your experimentation data has been ingested into Adobe Experience Platform, create a connection in Customer Journey Analytics to one or more experiment datasets.

Adobe’s statistical methodology

To provide easily interpretable and safe statistical inference, Adobe has adopted a statistical methodology based on Anytime Valid Confidence Sequences.

A confidence sequence is a sequential analog of a confidence interval. To understand what a confidence sequence is, imagine you repeat your experiments one hundred times. And calculate an estimate of the mean business metric (for example the open rate of an email) and its associated 95%-confidence sequence for every new user that enters the experiment.

A 95% confidence sequence includes the “true” value of the business metric in 95 out of the 100 experiments that you ran. (A 95% confidence interval could only be calculated once per experiment to give the same 95% coverage guarantee; not with every single new user). Confidence sequences therefore allow you to monitor experiments continuously, without increasing false positive error rates, that is, they allow “peeking” at results.

Interpret non-randomized dimensions

Customer Journey Analytics allows analysts to select any dimension as the experiment. But how do you interpret an analysis where the dimension chosen as the experiment is not one for which persons are randomized?

For example, consider an ad that a person sees. You may be interested in measuring the change in some metric (for example, average revenue) if you decide to show persons ad B instead of ad A. The causal effect of showing ad B, instead of ad A, is of central importance in arriving at the marketing decision. This causal effect may be measured as the average revenue over the whole population, if you replaced the status quo of showing ad A with the alternate strategy of showing ad B.

A/B testing is the gold standard within the industry for objectively measuring the effects of such interventions. The critical reason why an A/B test gives rise to a causal estimate is due to the randomization of persons to receive one of the possible variants.

Now, consider a dimension that is not achieved by randomization, for example, the US state of the person. Persons primarily come from two states, New York and California. The average revenue of sales of a winter clothing brand can be different in the two states due to the differences in the regional weather. In such a situation, the weather may be the true causal factor behind winter clothing sales, and not the fact that the geographical states of persons are different.

The experimentation panel in Customer Journey Analytics lets you analyze data as average revenue difference by states of the persons. In such a situation, the output does not have a causal interpretation. However, such an analysis may still be of interest. It provides an estimate (along with measures of uncertainty) of the difference in average revenue by states of the persons. This value is also referred to as Statistical Hypothesis Testing. The output of this analysis may be interesting, but not necessarily actionable. Simply because you have not randomized and sometimes cannot randomize persons to one of the possible values of the dimension.

The following illustration contrasts these situations:

When you want to measure the impact of intervention X on outcome Y, it is possible that the real cause of both is the confounding factor C. If the data is not achieved by randomizing persons on X, the impact is harder to measure, and the analysis explicitly accounts for C. Randomization breaks the dependence of X on C, allowing us to measure the effect of X on Y without having to worry about other variables.