Statistical calculations in A/Bn tests
This article documents the detailed statistical calculations used in manual A/Bn tests in Adobe Target. Definitions are provided for Conversion Rate, Confidence Interval of Conversion Rate, Lift, Confidence Interval for Lift, and Confidence.
Mean performance
The following section explains the calculations used in the previous illustration.
Conversion Rate and Revenue Per Visitor (RPV) Campaigns
The following illustration shows Conversion Rate, Confidence Interval of Conversion Rate, and the number of Conversions in a Target report. For example, the first line shows that for Experience A: the Conversion Rate is 25.81% with a Confidence Interval of ±7.7% and 32 conversions were recorded. Given that 124 Visitors saw the experience, this equates to 32/124 = 25.81%.
{width="25%"}
The conversion rate or mean, μ_{ν}, for each experience ν in an experiment is defined as a ratio of the sum of the metric to the number of units assigned to that metric, N_{ν}:
{width="125px"}
Here,

Y_{iν} is the value of the metric for each unit i, that has been assigned to a given experience ν.

The sum over units i depends on the choice of counting methodology.
 If Visitors is used as the counting methodology, each unit is a unique visitor defined as a unique participant in the activity for the life of the activity.
 If Visits is used as the counting methodology, each unit is a unique visit defined as a unique participant in an experience during a Target session (with a unique
sessionId
). When thesessionId
changes, or the visitor reaches the conversion step, a new visit is counted.  If Activity Impressions is used as the counting methodology, each unit is a unique impression defined as each time a visitor loads any page of the activity.
Confidence Interval of Mean/Conversion Rate
The confidence interval of the conversion rate is intuitively defined as range of possible conversion rates that is consistent with the underlying data.
When running experiments, the conversion rate for a given experience is an estimate of the “true” conversion rate. To quantify the uncertainty in this estimate, Target uses a confidence interval. Target always reports a 95% confidence interval, which means that in the end, 95% of confidence intervals calculated include the true conversion rate of the experience.
A 95% confidence interval of conversion rate μ_{ν} is defined as the range of values:
{width="30%"}
Where the standard error for the mean is defined as
{width="75px"}
Where an unbiased estimate of the sample standard deviation is used:
{width="200px"}
When the campaign is a conversion rate campaign (i.e., the conversion metric is binary), the standard error reduces to:
{width="150px"}
Lift
The following illustration shows Lift and Confidence Interval of Lift in a Target Report. The number represents the average of the range of the lift bounds, and the arrow reflects if the lift is positive or negative. The arrow displays in grey until the confidence passes 95%. After confidence passes the threshold, the arrow is green or red based on a positive or negative lift.
{width="35%"}
The lift between an experience ν, and the control experience ν_{0} is the relative “delta” in conversion rates, defined as
{width="15%"}
Where the individual conversion rates are as defined above. More simply,
Lift(Experience N) = (Performance_Experience_N  Performance_Control)/ Performance_Control
If the conversion rate of the control experience ν_{0} is 0, there is no lift.
Confidence Interval of Lift
The boxplot graph in the Average Lift and Confidence Interval column represents the average value and 95% Confidence Interval of Lift. The boxplot is grey when there is any overlap in the confidence interval of a given noncontrol experience with the confidence interval of control experience. The boxplot is green or red when the range of given experience’s confidence interval is above or below the confidence interval of control experience.
The standard error of the lift between an experience ν, and the control experience ν_{0} is defined as:
{width="35%"}
Then the 95% Confidence Interval of the lift is:
{width="40%"}
This calculation uses the “Delta” method, and is described in more detail in this document
Confidence
The last column shows the confidence in a Target report. The confidence of an experience is a probability (denoted as a percentage) of obtaining a result as extreme as the one that is observed, given the null hypothesis is true. In terms of pvalues, the confidence displayed is 1  pvalue. Intuitively, higher confidence means that it is less likely that the control and noncontrol experience have equal conversion rates.
In Target, a twotailed Welch’s ttest is performed between the test experience and the control experience to test if the means of test and control experiences are the same. Because we usually do not know if sample sizes and variances of two groups are the same before running the experiment, and Target also allows you to have unequal percentages of traffic sent to each experience, we do not assume that the variance for each experience is equal. Thus, Welch’s ttest is chosen instead of Student’s ttest.
To perform Welch’s ttest, we first start calculating the tstatistic and the degrees of freedom, then run a twotailed ttest to generate the pvalue. Finally, we calculate the confidence based on pvalue.
The tstatistic is defined to be the difference of the means of any two independent random variables, ν and ν_{0}, divided by the standard error of the difference:
{width="100px"}
Where μ_{v} and μ_{v0} are the means of ν and ν_{0} respectively, and the standard error of the difference between μ_{v} and μ_{v0} are given by:
{width="150px"}
Where σ^{2}_{v} and σ^{2}_{v0} are the variances of two experiences ν and ν_{0} respectively, and N_{v} and N_{v0} are sample sizes for ν and ν_{0} respectively.
For Welch’s ttest, the degree of freedom is calculated as following:
{width="180px"}
And degree of freedom for ν and ν_{0} are defined as:
{width="100px"}
{width="100px"}
Then the pvalue can be computed from the area in the tails of the tdistribution:
{width="20%"}
Finally, the confidence reported in Target is defined as:
{width="20%"}
Performing Calculations offline
The downloaded CSV report includes only raw data and does not include calculated metrics, such as revenue per visitor, lift, or confidence used for A/B tests.
To compute these statistical quantities, download the Target Complete Confidence Calculator Excel file to input the activity’s value.