Streaming ingestion overview

Streaming ingestion for Adobe Experience Platform provides users a method to send data from client and server-side devices to Experience Platform in real time.

What can you do with streaming ingestion?

Adobe Experience Platform enables you to drive coordinated, consistent, and relevant experiences by generating a Real-Time Customer Profile for each of your individual customers. Streaming ingestion plays a key role in building these profiles by enabling you to deliver Profile data into data lake with as little latency as possible.

The following video is designed to help support your understanding of streaming ingestion, and outlines the concepts above.

Stream profile records and ExperienceEvents

With streaming ingestion, users can stream profile records and ExperienceEvents to Experience Platform in seconds to help drive real-time personalization. All data sent to streaming ingestion APIs is automatically persisted in data lake.

Please read the create a streaming connection guide for more information.

Stream to datasets

Once you are confident that your data is clean, you can enable your datasets for Real-Time Customer Profile and Identity Service.

For more information on enabling a dataset for Profile and Identity Service, please read the configure a dataset guide.

What is the expected latency for streaming ingestion on Experience Platform?

IMPORTANT
Guardrails for streaming ingestion are bound to the total license usage entitlement that corresponds with your entire organization. Additionally, data usage in development sandboxes are limited to 10% of your total profiles. For more information about license usage entitlement, read the data management best practices guide. To learn how to set limits to your streaming throughput, read the Capacity overview.
Destination
Expected latency
Real-Time Customer Profile
  • < 15 minutes at the 95th percentile for B2C Data Ingestion.
  • < 30 minutes at the 95th percentile for B2B Data Ingestion.
Data lake
< 60 minutes

Request per seconds (RPS) guidance on streaming ingestion

The table below displays guidance on the request per seconds limits for streaming ingestion.

RPS limit
Notes
1000 requests per second
These can contain multiple messages when using /collection/batch endpoint.
10000 individual messages per second
The messages can be grouped into fewer actual requests when using the /collection/ endpoint.
IMPORTANT
The enforced limit becomes 60 requests per minute when using synchronous validation as it is intended for debugging purposes.

Adobe Experience Platform extension

You can use the Adobe Experience Platform extension to create a new streaming connection. The Experience Platform extension provides actions to send beacons formatted in Experience Data Model (XDM) for real-time ingestion to Experience Platform. Visit the Experience Platform Extension documentation for more information.

recommendation-more-help
experience-platform-help-ingestion