Calculate confidence for Auto-Allocate activities manually

Description description

Environment

Adobe Target

Issues/Symptoms

Confidence of Auto-Allocate activity is different in Target UI and when calculated through Confidence calculator excel. Why is that?

Resolution resolution

Auto-Allocate uses a different confidence calculator (based on Bernstein bounds) compared to regular A/B tests (which is where the calculator you used applies; that uses a t-test, that is, p-values derived from the t-distribution).

The confidence calculations used by Auto-Allocate are a lot more conservative than those used by regular t-tests. It is done to control false positives, that is, prevent us from making an early determination that one arm is better than another when the effect we’re seeing is due to randomness.

So that’s why classical t-test-based confidence is > 93%, but the Auto-Allocate confidence is much lower. It is also shared in the Auto-Allocate overviewdocuments.

A 60% confidence level is required for auto allocate activities as mentioned here.