Streaming ingestion overview
Last update: Tue Feb 13 2024 00:00:00 GMT+0000 (Coordinated Universal Time)
Streaming ingestion for Adobe Experience Platform provides users a method to send data from client and server-side devices to Experience Platform in real time.
What can you do with streaming ingestion?
Adobe Experience Platform enables you to drive coordinated, consistent, and relevant experiences by generating a Real-Time Customer Profile for each of your individual customers. Streaming ingestion plays a key role in building these profiles by enabling you to deliver Profile data into the Data Lake with as little latency as possible.
The following video is designed to help support your understanding of streaming ingestion, and outlines the concepts above.
In this video, I’m going to give you a quick introduction to streaming ingestion with Adobe Experience Platform. At Adobe, we believe people buy experiences not products, and to deliver powerful and compelling experiences, you need up-to-date information so you can make meaningful decisions in milliseconds. So what does this mean? It means providing the fastest path to update a real-time customer profile. It means being able to convert hundreds of thousands of signals into actionable insights as quickly as possible. Using Experience Platforms streaming ingestion, you can be sure that any data you send will be available in the real-time customer profile in under a second. This data can be captured on your website or mobile apps, CRM and ERP systems or from any other source which is able to communicate over HTTP or public cloud streaming infrastructure. Data sent through streaming to Experience Platform is stitched together with existing data in the real-time customer profile in real-time. You can then use this data for real-time segmentation and to activate other marketing processes.
With the Adobe Experience Platform extension in the launch interface, you can quickly configure your websites to send data directly to Experience Platform. If you’re sending personally identifiable information or other sensitive data to Platform, you can configure data collection to be secure and prevent data from untrusted sources.
To help make debugging easier, streaming ingestion provides two modes of validation, synchronous and asynchronous. In synchronous mode, streaming ingestion will drop records which fail XDM validation and include the reason for the failure. These failed records will not be sent downstream. Synchronous validation provides immediate feedback to help you maintain velocity as you figure out how to work with XDM formatted JSON payloads. In asynchronous mode, where data loss must be prevented, streaming validation will detect and move bad data to a separate location in the Experience Platform data lake. Once data starts flowing, you can monitor it in the platform interface. Experience Platform lets you observe incoming data all the way from collection to consumption in the data lake and real-time customer profile. All streaming ingested data lands in the Experience Platform data lake and is available with a maximum latency of 15 minutes. In parallel, the ingested data builds out identity graphs and the real-time customer profile. Both of these services consume the data in near real-time in under a second from the moment it was received. To learn more about streaming ingestion, head on over to the Adobe Experience Platform documentation. I hope you enjoyed this overview of streaming ingestion in Adobe Experience Platform.
Stream profile records and ExperienceEvents
With streaming ingestion, users can stream profile records and ExperienceEvents to Platform in seconds to help drive real-time personalization. All data sent to streaming ingestion APIs is automatically persisted in the Data Lake.
Please read the create a streaming connection guide for more information.
Stream to datasets
Once you are confident that your data is clean, you can enable your datasets for Real-Time Customer Profile and Identity Service.
For more information on enabling a dataset for Profile and Identity Service, please read the configure a dataset guide.
Destination
Expected latency
Real-Time Customer Profile
< 15 minutes at the 95th percentile
Request per seconds (RPS) guidance on streaming ingestion
The table below displays guidance on the request per seconds limits for streaming ingestion.
1000 requests per second
These can contain multiple messages when using /collection/batch endpoint.
10000 individual messages per second
The messages can be grouped into fewer actual requests when using the /collection/batch endpoint.
The enforced limit becomes 60 requests per minute when using synchronous validation as it is intended for debugging purposes.
You can use the Adobe Experience Platform extension to create a new streaming connection. The Experience Platform extension provides actions to send beacons formatted in Experience Data Model (XDM) for real-time ingestion to Experience Platform. Visit the Experience Platform Extension documentation for more information.