Dataflow ingestion from incorrect source folder in Adobe Experience Platform

Last update: March 25, 2025

This article provides a solution to address an issue where a dataflow configured to ingest data from Azure Blob Storage reads files from an incorrect folder. The misconfiguration leads to missed or duplicate events in the dataset, affecting downstream services like Data Lake and Customer Journey Analytics (CJA).

Description

Environment

Adobe Experience Platform (AEP)

Issue/Symptoms

A dataflow configured to ingest files from one Azure Blob Storage folder instead ingests from an incorrect folder. This issue results in:

  • Missed events in the intended dataset.
  • Duplicate events within datasets due to ingestion from the wrong source folder.

The problem typically arises in the following scenarios:

  1. A dataflow initially set up with one source path is later updated to a different source path using API calls.
  2. The system does not support updating the source connection for existing flows, causing continued ingestion from the original source path, despite UI indications of a successful update.

Resolution

  1. Create a new dataflow configured with the correct source path, as updates to existing dataflows source paths are not supported.
  2. Ensure that all configurations match your intended setup before initiating any new flow runs.
  3. Remove duplicates or incorrectly ingested records from your datasets by deleting prior batches. Refer to the “Delete a Batch” section in Adobe documentation: Batch API Overview – Revert Action. Use the action=REVERT query parameter to delete previous batches.
  4. Regularly monitor future flow runs to ensure they adhere strictly to their defined configurations without deviation.
  5. Refrain from attempting updates to the source path of an existing dataflow using the API or UI, as this is unsupported and may lead to ingestion errors.
  6. Monitor Adobe release notes and updates for any changes to platform capabilities related to dynamic configuration changes. Updates can be found at: Adobe Experience Platform Release Notes.
recommendation-more-help