Send partial row updates to Real-Time Customer Profile using Data Prep

Use streaming upserts in Data Prep to send partial row updates to Real-Time Customer Profile data while also creating and establishing new identity links with a single API request.

By streaming upserts, you can retain the format of your data while translating that data to Real-Time Customer Profile PATCH requests during ingestion. Based on the inputs you provide, Data Prep allows you to send a single API payload and translate the data to both Real-Time Customer Profile PATCH and Identity Service CREATE requests.

Data Prep uses header parameters to distinguish between inserts and upserts. All rows that use upserts must have a header. You can use upserts with or without identity descriptors. If you are using upserts with identities, you must follow the configuration steps outlined in the section on configuring the identity dataset. If you are using upserts without identities, then you do not need to provide identity configurations in your request. Read the section on streaming upserts without identities for more information.

This document provides information on how to stream upserts in Data Prep.

Getting started

This overview requires a working understanding of the following components of Adobe Experience Platform:

Use streaming upserts in Data Prep streaming-upserts-in-data-prep

Streaming upserts high-level workflow

Streaming upserts in Data Prep works as follows:

Configure the identity dataset configure-the-identity-dataset

If new identities must be linked, then you must create and pass an additional dataset in the incoming payload. When creating an identity dataset, you must ensure that the following requirements are met:

Required fields in the schemas associated with the identity dataset identity-dataset-required-fileds

If your schema contains required fields, validation of the dataset must be suppressed in order to enable Identity Service to only receive the identities. You can suppress validation by applying the disabled value to the acp_validationContext parameter. See the example below:

curl -X POST 'https://platform.adobe.io/data/foundation/catalog/dataSets/62257bef7a75461948ebcaaa' \
  -H 'Authorization: Bearer {ACCESS_TOKEN}' \
  -H 'Content-Type: application/json' \
  -H 'x-api-key: {API_KEY}' \
  -H 'x-gw-ims-org-id: {IMS_ORG}' \
  -H 'x-sandbox-name: {SANDBOX_NAME}' \
  -d '{
    "tags": {
        "acp_validationContext": [
            "disabled"
        ],
        "unifiedProfile": [
            "enabled:false"
        ],
        "unifiedIdentity": [
            "enabled:true"
        ]
    }
}'

Incoming payload structure

The following displays an example of an incoming payload structure that establishes new identity links.

Payload with identity configuration

{
  "header": {
    "flowId": "923e2ac3-3869-46ec-9e6f-7012c4e23f69",
    "imsOrgId": "{ORG_ID}",
    "datasetId": "621fc19ab33d941949af16c8",
    "operations": {
        "data": "create" (default)/"merge"/"delete",
        "identity": "create",
        "identityDatasetId": "621fc19ab33d941949af16d9"
    }
  }
... //The raw data attributes are included here as the key/value pairs of the "body" property.
}

Supported operations

The following operations are supported by Real-Time Customer Profile:

The following operations are supported by Identity Service:

Payload without identity configuration payload-without-identity-configuration

If new identities do not need to be linked, then you can omit the identity and identityDatasetId parameters in the operations. Doing so sends data only to Real-Time Customer Profile and skips the Identity Service. See the payload below for an example:

{
  "header": {
    "flowId": "923e2ac3-3869-46ec-9e6f-7012c4e23f69",
    "imsOrgId": "{ORG_ID}",
    "datasetId": "621fc19ab33d941949af16c8",
    "operations": {
        "data": "create"/"merge"/"delete",
    }
  }
... //The raw data attributes are included here as the key/value pairs of the "body" property.
}

Dynamically pass primary identities

For XDM updates, the schema must be enabled for Profile and contain a primary identity. You can specify the primary identity of an XDM schema in two ways:

Designate a static field as the primary identity field in the XDM schema

In the example below, state, homePhone.number and other attributes are upserted with their respective given values into the Profile with the primary identity of sampleEmail@gmail.com. An XDM entity update message is then generated by the streaming Data Prep component. Real-Time Customer Profile then confirms that XDM update message to upsert the profile record.

curl -X POST 'https://dcs.adobedc.net/collection/9aba816d350a69c4abbd283eb5818ec3583275ffce4880ffc482be5a9d810c4b' \
  -H 'Content-Type: application/json' \
  -H 'x-adobe-flow-id: d5262d48-0f47-4949-be6d-795f06933527' \
  -d '{
    "header": {
        "flowId" : "d5262d48-0f47-4949-be6d-795f06933527",
        "imsOrgId": "{ORG_ID}",
        "datasetId": "62259f817f62d71947929a7b",
        "operations": {
         "data": "create"
     }
    },
    {
        "body": {
        "homeAddress": {
            "country": "US",
            "state": "GA",
            "region": "va7"
        },
        "homePhone": {
            "number": "123.456.799"
        },
        "identityMap": {
            "Email": [{
                "id": "sampleEmail@gmail.com",
                "primary": true
            }]
        },
      "personalEmail": {
            "address": "sampleEmail@gmail.com",
            "primary": true
       },
      "personID": "346576345",
      "_id": "346576345",
      "timestamp": "2021-05-05T17:51:45.1880+02",
      "workEmail": "sampleWorkEmail@gmail.com"
  }
}'

Designate one of the identity fields as the primary identity through the identity map field group in the XDM schema

In this example, the header contains the operations attribute with the identity and identityDatasetId properties. This allows data to be merged with Real-Time Customer Profile and also for identities to be passed to Identity Service.

curl -X POST 'https://dcs.adobedc.net/collection/9aba816d350a69c4abbd283eb5818ec3583275ffce4880ffc482be5a9d810c4b' \
  -H 'Content-Type: application/json' \
  -H 'x-adobe-flow-id: d5262d48-0f47-4949-be6d-795f06933527' \
  -d '{
    "header": {
        "flowId" : "d5262d48-0f47-4949-be6d-795f06933527",
        "imsOrgId": "{ORG_ID}",
        "datasetId": "62259f817f62d71947929a7b",
        "operations": {
            "data": "merge",
            "identity": "create",
            "identityDatasetId": "6254a93b851ecd194b64af9e"
      }
    },
    {
       "body": {
        "homeAddress": {
            "country": "US",
            "state": "GA",
            "region": "va7"
        },
        "homePhone": {
            "number": "123.456.799"
        },
        "identityMap": {
            "Email": [{
                "id": "sampleEmail@gmail.com",
                "primary": true
            }]
        },
      "personalEmail": {
            "address": "sampleEmail@gmail.com",
            "primary": true
       },
      "personID": "346576345",
      "_id": "346576345",
      "timestamp": "2021-05-05T17:51:45.1880+02",
      "workEmail": "sampleWorkEmail@gmail.com"
  }
 }'

Known limitations and key considerations

The following outlines a list of known limitations to consider when streaming upserts with Data Prep:

Next steps

By reading this document, you should now understand how to stream upserts in Data Prep to send partial row updates to your Real-Time Customer Profile data, while also creating and linking identities with a single API request. For more information on other Data Prep features, please read the Data Prep overview. To learn how to use mapping sets within the Data Prep API, please read the Data Prep developer guide.