Data preparation and ingestion blueprint

Last update: January 10, 2025

CREATED FOR:

  • Developer
  • User

Data preparation and ingestion Blueprint encompasses all the methods by which data can be prepared and ingested into Adobe Experience Platform.

Data preparation includes the mapping of source data to Experience Data Model (XDM) schema. It also includes performing transformations on data, including date formatting, field splitting/concatenation/conversions, and joining/merging/re-keying of records. Data preparation helps unify customer data to provide aggregated/filtered analysis, including reporting or preparing data for customer profile assembly/data science/activation.

Architecture

Reference architecture for the Data Preparation and Ingestion Blueprint

Data ingestion guardrails

The below diagram illustrates the average performance guardrails and latency for data ingestion into Adobe Experience Platform.

Experience Platform Data Flow

Data ingestion methods

Streaming Sources
MethodCommon Use CasesProtocolsConsiderations
Adobe Web/Mobile SDK
  • Data collection from websites and mobile apps.
  • Preferred method for client side collection.
Push, HTTP, JSON
  • Implement multiple Adobe applications leveraging a single SDK.
HTTP API Connector
  • Collection from streaming sources, transactions, relevant customer events and signals
Push, REST API, JSON
  • Data is streamed directly to the hub so no real-time Edge segmentation or event forwarding.
Edge Network API
  • Collection from streaming sources, transactions, relevant customer events and signals from the globally distributed Edge Network
Push, REST API, JSON
  • Data is streamed through the Edge Network. Support for real-time segmentation on the Edge.
Adobe Applications
  • Prior implementation of Adobe Analytics, Marketo, Campaign, Target, AAM
Push, Source Connectors and API
  • Recommended approach is migration to Web/Mobile SDK over traditional application SDKs.
Streaming Source Connectors
  • Ingestion of a enterprise event stream, typically used for sharing enterprise data to multiple down-stream applications.
Push, REST API, JSON
  • Must be streamed in XDM format.
Streaming Sources SDK
  • Similar to HTTP API Connector, allows self-service configuration card of a external data stream.
Push, HTTP API, JSON
  • Edge Network
Batch Sources
MethodCommon Use CasesProtocolsConsiderations
Batch Ingestion API
  • Ingestion from a enterprise managed que. Cleansing and transformation of data prior to ingestion.
Push, JSON or Parquet
  • Must manage batches and files for ingestion
Batch Source Connectors
  • Common approach for ingestion of files from cloud storage locations.
  • Connectors to common CRM and marketing applications.
  • Ideal for ingesting large amounts of historical data.
Pull, CSV, JSON, Parquet
  • Not always on, immediate ingestion.
Data Landing Zone
  • Adobe provisioned file storage location to push files to for ingestion.
Push, CSV, JSON, Parquet
Batch Sources SDK
  • Allows self-service configuration card of an external data source.
  • Ideal for partner connectors or for a tailored workflow experience for setting up a enterprise connector.
Pull, REST API, CSV or JSON Files
  • Examples: MailChimp, One Trust, Zendesk
Previous pageMulti-sandbox event-forwarding data collection
Next pageData access & export

Data Collection