Resolve duplicated data in Adobe Experience Platform schema fields
If you encounter duplicated entries in Adobe Experience Platform schema fields, you can disable the upsert feature to ensure full attribute replacement and avoid duplication.
Description description
Environment
Adobe Experience Platform (AEP)
This issue occurs with schema fields with string arrays marked as Unique items only.
Issue/Symptoms
When ingesting data into AEP from external sources such as AWS, profiles experience duplicated entries within array fields. This duplication arises because despite having the Unique items only option enabled for these arrays:
- Datasets are configured with the
upsertfeature, which appends to the array rather than replacing existing entries. - Identical newsletter IDs and interest IDs can be added to user profiles each time they are updated.
Resolution resolution
Note: Always test changes in a development sandbox environment before applying them to production systems.
To address this issue of duplicate entries in profile schemas:
- Consider disabling the
upsertfunctionality on your dataset. Theupsertfeature, which stands for update or insert allows new data to be added or existing data to be updated within a dataset. - Ensure that every update includes all relevant attributes, not just the ones being changed, to replace the entire array rather than appending to it.
- If you have multiple datasets referring to the same individual profile schema (for example, one for interests/newsletters and another for cookie IDs/consent preferences), consider storing different types of information in separate schemas to avoid conflicts during data merging.
Related reading
Define array fields in the UI in the Experience Data Model (XDM) Guide