DocumentationExperience PlatformData Ingestion Guide

Streaming ingestion overview

Last update: Thu May 30 2024 00:00:00 GMT+0000 (Coordinated Universal Time)

CREATED FOR:

  • Developer

Streaming ingestion for Adobe Experience Platform provides users a method to send data from client and server-side devices to Experience Platform in real time.

What can you do with streaming ingestion?

Adobe Experience Platform enables you to drive coordinated, consistent, and relevant experiences by generating a Real-Time Customer Profile for each of your individual customers. Streaming ingestion plays a key role in building these profiles by enabling you to deliver Profile data into the Data Lake with as little latency as possible.

The following video is designed to help support your understanding of streaming ingestion, and outlines the concepts above.

https://video.tv.adobe.com/v/28425?quality=12&learn=on

Transcript
In this video, I’m going to give you a quick introduction to streaming ingestion with Adobe Experience platform. At Adobe, we believe people by experiences, not products. And to deliver powerful and compelling experiences, you need up to date information so you can make meaningful decisions in milliseconds. So what does this mean? It means providing the fastest path to update a real time customer profile. It means being able to convert hundreds of thousands of signals into actionable insights as quickly as possible. Using experience platforms, streaming, ingestion. You can be sure that any data you send will be available in the real time customer profile. This data can be captured on your website or mobile apps, CRM and ERP systems, or from any other source which is able to communicate over Http or public cloud streaming infrastructure. Data sent through streaming to experienced platform is stitched together with existing data and the real time customer profile in real time. You can then use this data for real time segmentation and to activate other marketing processes with the Adobe Experience Platform extension in the launch interface. You can quickly configure your websites to send data directly to experience Platform. If you’re sending personally identifiable information or other sensitive data to platform, you can configure data collection to be secure and prevent data from untrusted sources.
To help make debugging easier. Streaming. Ingestion provides two modes of validation. Synchronous and asynchronous. In synchronous mode, streaming ingestion will drop records which fail zdm validation and include the reason for the failure. These failed records will not be sent downstream. Synchronous validation provides immediate feedback to help you maintain velocity as you figure out how to work with Xdm formatted Json payloads.
An asynchronous mode where data loss must be prevented. Streaming validation will detect and move bad data to a separate location in the experience platform. Data Lake.
Once data starts flowing, you can monitor it in the platform interface. Experience platform lets you observe incoming data all the way from collection to consumption in the data lake and real time customer profile. All streaming ingested data lands in the Experience Platform Data Lake, and is available with a maximum latency of 15 minutes.
In parallel, the ingested data builds out identity graphs and the real time customer profile. To learn more about streaming ingestion, head on over to the Adobe Experience Platform documentation. I hope you enjoyed this overview of streaming, ingestion and Adobe Experience platform.

Stream profile records and ExperienceEvents

With streaming ingestion, users can stream profile records and ExperienceEvents to Platform in seconds to help drive real-time personalization. All data sent to streaming ingestion APIs is automatically persisted in the Data Lake.

Please read the create a streaming connection guide for more information.

Stream to datasets

Once you are confident that your data is clean, you can enable your datasets for Real-Time Customer Profile and Identity Service.

For more information on enabling a dataset for Profile and Identity Service, please read the configure a dataset guide.

What is the expected latency for streaming ingestion on Experience Platform?

IMPORTANT
Guardrails for streaming ingestion are calculated at the organization level and not the sandbox level. This means that your data usage per sandbox is bound to the total license usage entitlement that corresponds with your entire organization. Additionally, data usage in development sandboxes are limited to 10% of your total profiles. For more information about license usage entitlement, read the data management best practices guide.
Destination
Expected latency
Real-Time Customer Profile
< 15 minutes at the 95th percentile
Data lake
< 60 minutes

Request per seconds (RPS) guidance on streaming ingestion

The table below displays guidance on the request per seconds limits for streaming ingestion.

RPS limit
Notes
1000 requests per second
These can contain multiple messages when using /collection/batch endpoint.
10000 individual messages per second
The messages can be grouped into fewer actual requests when using the /collection/batch endpoint.
IMPORTANT
The enforced limit becomes 60 requests per minute when using synchronous validation as it is intended for debugging purposes.

Adobe Experience Platform extension

You can use the Adobe Experience Platform extension to create a new streaming connection. The Experience Platform extension provides actions to send beacons formatted in Experience Data Model (XDM) for real-time ingestion to Experience Platform. Visit the Experience Platform Extension documentation for more information.

recommendation-more-help
2ee14710-6ba4-4feb-9f79-0aad73102a9a