The Analytics for Target (A4T) integration for Auto-Target activities uses Adobe Target’s ensemble machine learning (ML) algorithms to choose the best experience for each visitor based on their profile, behavior, and context, all while using an Adobe Analytics goal metric.
While rich analysis capabilities are available in Adobe Analytics Analysis Workspace, a few modifications to the default Analytics for Target panel are required to correctly interpret Auto-Target activities, due to differences between experimentation activities (manual A/B and Auto-Allocate) and personalization activities (Auto-Target).
This tutorial walks through the recommended modifications for analyzing Auto-Target activities in Workspace, which are based on the following key concepts:
To create an A4T for Auto-Target report, either start with the Analytics for Target panel in Workspace, as shown below, or begin with a freeform table. Then make the following selections:
Figure 1: Analytics for Target panel setup for Auto-Target activities.
To set up your Analytics for Target panel for Auto-Target activities, choose any control experience, choose Visits as the normalizing metric, and choose the same goal metric that was chosen for optimization during Target activity creation.
The default A4T panel is designed for classic (manual) A/B tests or Auto-Allocate activities where the goal is to compare the performance of individual experiences against the Control experience. In Auto-Target activities, however, the first order comparison should be between the Control strategy and the Targeted strategy (in other words, determining the lift of the overall performance of the Auto-Target ensemble ML model over the Control strategy).
To perform this comparison, use the Control vs Targeted (Analytics for Target) dimension. Drag and drop to replace the Target Experiences dimension in the default A4T report.
Note this replacement invalidates the default Lift and Confidence calculations on the A4T panel. To avoid confusion, you can remove these metrics from the default panel, leaving the following report:
Figure 2: The recommended baseline report for Auto-Target activities. This report has been configured to compare Targeted traffic (served by the ensemble ML model) against your Control traffic.
Currently, Lift and Confidence numbers are not available for Control vs Targeted dimensions for A4T reports for Auto-Target. Until support is added, Lift and Confidence can be computed manually by downloading the confidence calculator.
To gain further insight into how the ensemble ML model is performing, you may examine Experience-level breakdowns of the Control vs Targeted dimension. In Workspace, drag the Target Experiences dimension onto your report, then break down each of the Control and Targeted dimensions separately.
Figure 3: Breaking down the Targeted dimension by Target Experiences
An example of the resulting report is shown here.
Figure 4: A standard Auto-Target report with Experience-level breakdowns. Note your goal metric may be different, and your Control strategy may have a single experience.
In Workspace, click the gear icon to hide the Percentages in the Conversion Rate column, to help keep the focus on the experience conversion rates. Note the conversion rates will then be formatted as decimals, but interpret them as percentages accordingly.
When analyzing an Auto-Target activity, always choose Visits as the default normalizing metric. Auto-Target personalization selects an experience for a visitor once per visit (formally, once per Adobe Target session), which means the experience shown to a user can change on every single visit. Thus, if you use Unique Visitors as the normalizing metric, the fact that a single user may end up seeing multiple experiences (across different visits) would lead to confusing conversion rates.
A simple example demonstrates this point: consider a scenario in which two visitors enter a campaign which has only two experiences. The first visitor visits twice. They are assigned to Experience A on the first visit, but Experience B on the second visit (due to their profile state changing on that second visit). After the second visit, the visitor converts by placing an order. The conversion is attributed to the most recently shown experience (Experience B). The second visitor also visits twice, and is shown Experience B both times, but never converts.
Let us compare visitor-level and visit-level reports:
|Experience||Unique Visitors||Visits||Conversions||Visitor norm. Conv. Rate||Visit norm. Conv. Rate|
Table 1: Example comparing visitor-normalized and visit-normalized reports for a scenario in which decisions are sticky to a visit (and not visitor, as with regular A/B testing). Visitor-normalized metrics are confusing in this scenario.
As shown in the table, there is a clear incongruence of visitor-level numbers. Despite the fact there are two total unique visitors, this is not a sum of individual unique visitors to each experience. While the visitor-level conversion rate is not necessarily wrong, when one compares individual experiences, visit-level conversion rates arguably make much more sense. Formally, the unit of analysis (“visits”) is the same as the unit of decision stickiness, which means that experience-level breakdowns of metrics may be added and compared.
Adobe Analytics’ default counting methodology for visits to a Target activity may include visits where the user did not interact with the Target activity. This is due to the way Target activity assignments are persisted in the Analytics visitor context. As a result, the number of visits to the Target activity can sometimes be inflated, resulting in a depression of conversion rates.
If you would prefer to report on visits where the user actually interacted with the Auto-Target activity (either through entry to the activity, a display/visit event, or a conversion), you can:
To create the segment:
Figure 5: Use a segment such as the one shown here to filter the Visits metric in your A4T for Auto-Target report
Once the segment has been created, use it to filter the Visits metric, so the Visits metric only includes visits where the user interacted with the Target activity.
To filter Visits using this segment:
The final panel will appear as follows.
Figure 6: Reporting panel with the “Hit with specific Auto-Target Activity” segment applied to the Visits metric. This ensures only visits where a user actually interacted with the Target activity in question are included in the report.
The A4T integration allows Auto-Target’s ML model to be trained using the same conversion event data that Adobe Analytics uses to generate performance reports. However, there are certain assumptions which must be employed in interpreting this data when training the ML models, which differ from the default assumptions made during the reporting phase in Adobe Analytics.
Specifically, Adobe Target’s ML models use a visit-scoped attribution model. That is, they assume a conversion must happen in the same visit as a display of content for the activity, in order for the conversion to be “attributed” to the decision made by the ML model. This is required for Target to guarantee timely training of its models; Target cannot wait for up to 30 days for a conversion (the default attribution window for reports in Adobe Analytics), before including it in the training data for its models.
Thus, the difference between the attribution used by Target’s models (during training) versus the default attribution used in querying data (during report generation) may lead to discrepancies. It may even appear that the ML models are performing poorly, when in fact the issue lies with attribution.
If the ML models are optimizing for a metric that is attributed differently from that of the metrics you are viewing in a report, the models may not perform as expected! To avoid this, ensure the goal metrics on your report use the same attribution used by Target’s ML models.
To view goal metrics that have the same attribution methodology used by Adobe Target’s ML models, follow these steps:
These steps ensure your report will attribute the goal metric to the display of the experience, if the goal metric event happened any time (“participation”) in the same visit that an experience was shown.
With the modifications to the Visit and goal metrics in preceding sections, the final modification you should make to your default A4T for Auto-Target reporting panel is to create conversion rates that are the correct ratio—that of a goal metric with the right attribution, to an appropriately filtered “Visits” metric.
Do this by creating a Calculated Metric using the following steps:
The complete calculated metric definition is shown here.
Figure 7: The visit- and attribution-corrected model conversion rate metric definition. (Note this metric is dependent on your goal metric and activity. In other words, this metric definition is not re-usable across activities.)
The Conversion rate metric from the A4T panel is not linked to the conversion event or the normalizing metric in the table. When you make the modifications suggested in this tutorial, the Conversion rate does not automatically adapt to the changes. Therefore, if you make the modification to one (or both) the conversion event attribution and the normalizing metric, then you must remember as a final step to also modify the Conversion rate, as shown above.
Combining all of the steps above into a single panel, the figure below shows a complete view of the recommended report for Auto-Target A4T activities. This report is the same as that used by Target’s machine learning models to optimize your goal metric, and it incorporates all the nuances and recommendations discussed in this tutorial. This report is also closest to the counting methodologies used in traditional Target-reporting driven Auto-Target activities.
Figure 8: The final A4T Auto-Target report in Adobe Analytics Workspace, which combines all the adjustments to metric definitions described in the previous sections of this document.