Monitor dataflows for destinations in the UI

Destinations allow you to activate your data from Adobe Experience Platform to countless external partners. Platform makes the process of tracking the flow of data to your destinations easier by providing transparency with dataflows.

The monitoring dashboard provides you with a visual representation of the journey of a dataflow, including the destination the data is activated to. This tutorial provides instructions on how you can either monitor dataflows directly in the destinations workspace or use the monitoring dashboard to monitor dataflows for your destinations using the Experience Platform user interface.

Getting started

This guide requires a working understanding of the following components of Adobe Experience Platform:

• Dataflows: Dataflows are a representation of data jobs that move data across Platform. Dataflows are configured across different services, helping move data from source connectors to target datasets, to Identity and Profile, and to Destinations.
  • Dataflow runs: Dataflow runs are the recurring scheduled jobs based on the frequency configuration of selected dataflows.
• Destinations: Destinations are pre-built integrations with commonly used applications that allow for the seamless activation of data from Platform for cross-channel marketing campaigns, email campaigns, targeted advertising, and many other use cases.
• Sandboxes: Experience Platform provides virtual sandboxes which partition a single Platform instance into separate virtual environments to help develop and evolve digital experience applications.

Monitor dataflows in the Destinations workspace

In the Destinations workspace within the Platform UI, navigate to the Browse tab and select the name of a destination that you want to view.

A list of existing dataflows appears. On this page is a list of viewable dataflows, including information about their destination, username, number of dataflows, and status.

See the following table for more information on statuses:

Dataflow runs for streaming destinations

For streaming destinations, the Dataflow runs tab provides an hourly update for metric data on your dataflow runs. The most prominent statistics labelled are for identities.

Identities represent the different facets of a profile. For example, if a profile contains both a phone number and an email address, that profile will have two identities.

A list of individual runs and their particular metrics is displayed, along with the following totals for identities:

• Identities activated: The total count of profile identities that were created or updated for activation.
• Identities excluded: The total number of profile identities that are skipped for activation based on missing attributes and consent violation.
• Identities failed: The total number of profile identities that are not activated to the destination due to errors.

Each individual dataflow run shows the following details:

• Dataflow run start: The time that the dataflow run started at.
• Processing time: The amount of time that it took for the dataflow to process.
• Profiles received: The total number of profiles received in the dataflow.
• Identities activated: The total number of profile identities that were successfully activated to the selected destination.
• Identities excluded: The total number of profile identities that are excluded for activation based on missing attributes and consent violation.
• Identities failed The total number of profile identities that are not activated to the destination due to errors.
• Activation rate: The percentage of received identities that have either been successfully activated or skipped. The following formula demonstrates how this value is calculated:
  
• Status: Represents the state the dataflow is in: either Completed or Processing. Completed means that all the identities for the corresponding dataflow run were exported within the one-hour period. Processing means that the dataflow run has not yet finished.

To view the details of a particular dataflow run, select the run’s start time from the list.

The details page for a dataflow run contains additional information such as the number of profiles received, the number of identities activated, the number of identities failed, and the number of identities excluded.

The details page also displays a list of identities that failed and identities that were excluded. Information for both the failed and excluded identities is displayed, including the error code, identity count, and description. By default, the list displays the failed identities. To show skipped identities, select the Identities excluded toggle.

Dataflow runs for batch destinations

For batch destinations, the Dataflow runs tab provides metric data on your dataflow runs. A list of individual runs and their particular metrics is displayed, along with the following totals for identities:

• Identities activated: The count of individual profile identities successfully activated to the selected destination.
• Identities excluded: The count of individual profile identities excluded for activation for the selected destination, based on missing attributes and consent violation.

Each individual dataflow run shows the following details:

• Dataflow run start: The time that the dataflow run started at.
• Processing time: The amount of time it took for the dataflow run to be processed.
• Profiles received: The total number of profiles received in the dataflow. This value is updated every 60 minutes.
• Identities activated: The total number of profile identities that were successfully activated to the selected destination.
• Identities excluded: The total number of profile identities that are excluded for activation based on missing attributes and consent violation.
• Status: Represents the state the dataflow is in. This can be one of three states: Success, Failed, and Processing. Success means that the dataflow is active and is exporting data according to its provided schedule. Failed means that the activation of data has been suspended due to errors. Processing means that the dataflow is not yet active and is generally encountered when a new dataflow is created.

To view details of a specific dataflow run, select the run’s start time from the list.

The details page for a dataflow, in addition to the details shown on the dataflows list, displays more specific information about the dataflow:

• Size of data: The size of the dataflow that is being exported.
• Total files: The total number of files exported in the dataflow.
• Last updated: The time the dataflow run was last updated.

The details page also displays a list of identities that failed and identities that were excluded. Information for both the failed and excluded identities is displayed, including the error code and description. By default, the list displays the failed identities. To show excluded identities, select the Identities excluded toggle.

Monitoring Destinations dashboard

To access the Monitoring dashboard, select Monitoring () in the left navigation. Once on the Monitoring page, select Destinations. The Monitoring dashboard contains metrics and information on the destination run jobs.

At the center of the dashboard is the Activation panel, which contains metrics and graphs that display data on the activation rate of the data which is exported to destinations.

By default, the data displayed contains the activation rates from the last 24 hours. Select Last 24 hours to adjust the time frame of records displayed. Available options include Last 24 hours, Last 7 days, and Last 30 days. Alternatively, you can select the dates on the calendar pop-up window that appears. Once you have selected dates, select Apply to adjust the time frame of the information shown.

The graph is displayed by default and you can disable it to expand the list of destinations below. Select the Metrics and graphs toggle to disable the graphs.

The Activation panel displays a list of destinations that contain at least one existing account. This list also includes information on the profiles received, profile records activated, profile records failed, profile records skipped, total failed dataflows, and the last updated date for these destinations.

You can also filter your list of destinations to only display the selected category of destinations. Select the My destinations dropdown, and select the destination type you want to filter to.

Additionally, you can enter a destination into the search bar to isolate to a single destination. If you want to see the destination’s dataflows you can select the filter beside it to see a list of its active dataflows.

If you want to view all existing dataflows across all destinations, select Dataflows.

A list of dataflows appears, grouped on a per-destination basis. You can see additional details for a specific dataflow by locating the destination you want to monitor, selecting the filter beside it, and then subsequently selecting the filter beside the dataflow you want more information about.

The dataflow runs page displays information on your dataflow runs, including the dataflow run start time, processing time, profiles received, identities activated, identities excluded, identities failed, activation rate, and status. To see more details about a specific dataflow run, select the filter beside the dataflow run start time.

The dataflows details page, in addition to the details shown on the dataflows list, displays more specific information about the dataflow:

• Dataflow run ID: The ID of the dataflow.
• IMS org ID: The IMS organization the dataflow belongs to.
• Last updated: The time the dataflow run was last updated.

The details page also displays a list of identities that failed and identities that were excluded. Information for both the failed and excluded identities is displayed, including the error code, identity count, and description. By default, the list displays the failed identities. To show skipped identities, select the Identities excluded toggle.

Next steps

By following this guide, you now know how to monitor dataflows for both batch and streaming destinations, including all the relevant information such as processing time, activation rate, and status. To learn more about dataflows in Platform, please read the dataflows overview. To learn more about destinations, please read the destinations overview.