Stream data to AEP
Adobe Experience Platform allows for both profile and experience events to be streamed and available in near real-time. All data sent to AEP via streaming is persisted in the data lake. Data can be streamed to existing data sets or to entirely new data sets via APIs or using Adobe Launch.
This article will cover the following:
- Streaming to the XDM Individual Profile
- Streaming to the XDM ExperienceEvent
- Using AEP the Launch extension to stream
The Postman collection is referenced throughout the article using the associated calls by number. More details on installing and using the Postman collection are available on the Github README page. There are also sample datasets of loyalty and profile data.
Prerequisites
- Authenticate to the platform.
- Gather the values for required headers from the authentication tutorial linked above.
Create a streaming connection
To stream to AEP you must first create a streaming connection. Streaming connections contain attributes like the source of streaming data and whether or not you are sending in records that belong to the Experience Data Model (XDM) schemas. After creating a streaming connection, you are given a unique URL that you use to stream data into AEP.
Go here for instructions on how to create a streaming connection via API or here for instructions on how to create a streaming connection via the UI.
curl -X POST https://platform.adobe.io/data/foundation/flowservice/connections \
-H 'Authorization: Bearer {ACCESS_TOKEN}' \
-H 'Content-Type: application/json' \
-H 'x-gw-ims-org-id: {IMS_ORG}' \
-H 'x-api-key: {API_KEY}' \
-H 'x-sandbox-name: {SANDBOX_NAME}' \
-d '{
"name": "Sample name",
"providerId": "521eee4d-8cbe-4906-bb48-fb6bd4450033",
"description": "Sample description",
"connectionSpec": {
"id": "bc7b00d6-623a-4dfc-9fdb-f1240aeadaeb",
"version": "1.0"
},
"auth": {
"specName": "Streaming Connection",
"params": {
"sourceId": "Sample connection",
"dataType": "xdm",
"name": "Sample connection"
}
}
}
Response:
{
"id": "77a05521-91d6-451c-a055-2191d6851c34",
"etag": "\"a500e689-0000-0200-0000-5e31df730000\""
}
Be sure to save the ID provided in the response above for future streaming ingestion calls (the Postman collection will save this for you in the CONNECTION_ID environment variable).
Stream profile data to AEP
For this section, use Postman call folders: 3: Real-time import, 3a: Real-time import for PROFILE data.
Detailed JSON requests with responses for streaming profile data are documented here.
Steps:
- Create an XDM Individual Profile Schema
- Set the Primary Identity Descriptor for XDM Individual Profile (primary key)
- Create a dataset for XDM Individual Profile Records
- Call the Streaming Ingestion APIs to create an XDM Individual Profile Record
- Retrieve the newly created profile
Stream experience events to AEP
For this section, use Postman call folders: 3: Real-time import, 3b: Real-time import for PROFILE data.
Detailed JSON requests with responses for streaming experience data are documented here.
Steps:
- Create an XDM ExperienceEvent Schema
- Set the Primary Identity Descriptor for XDM ExperienceEvent (primary key)
- Create a dataset for XDM ExperienceEvents
- Call the Streaming Ingestion APIs to create an XDM ExperienceEvent
- Retrieve the newly created event
Use Experience Platform tags to stream to AEP
The Adobe Experience Platform Launch extension provides a way to stream to AEP via Launch. To learn more, see this guide.