Data Preparation and Ingestion Blueprint

Data Preparation and Ingestion Blueprint encompasses all the methods by which data can be prepared and ingested into Adobe Experience Platform.

Data preparation includes the mapping of source data to Experience Data Model (XDM) schema. It also includes performing transformations on data, including date formatting, field splitting/concatenation/conversions, and joining/merging/re-keying of records. Data preparation helps unify customer data to provide aggregated/filtered analysis, including reporting or preparing data for customer profile assembly/data science/activation.

Architecture

Reference architecture for the Data Preparation and Ingestion Blueprint

Data Ingestion Methods

Methods of Ingestion Description
Web/Mobile SDK Latency:
  • Real time - same page collection to Edge Network
  • Streaming ingestion to Profile ~1 minute
  • Streaming ingestion to data lake (micro batch ~15 minutes)
Documentation:
Streaming Sources Latency:
  • Real time - same page collection to Edge Network
  • Streaming ingestion to Profile ~1 minute
  • Streaming ingestion to data lake (micro batch ~15 minutes)
Documentation
Streaming API Latency:
  • Real time - same page collection to Edge Network
  • Streaming ingestion to Profile ~1 minute
  • Streaming ingestion to data lake (micro batch ~15 minutes)
  • 7 GB/hour
Documentation
ETL Tooling Use ETL tools to modify and transform enterprise data before ingestion into Experience Platform.

Latency:
  • Timing dependent on external ETL tool scheduling, then standard ingestion guardrails apply based on the method used for ingestion.
Batch Sources Scheduled fetch from sources
Latency: ~ 200 GB/hour

Documentation
Video Tutorials
Batch API Latency:
  • Batch ingestion to Profile dependent on size and traffic loads ~45 minutes
  • Batch ingestion to data lake dependent on size and traffic loads
Documentation
Adobe Application Connectors Automatically ingest data that is sourced from Adobe Experience Cloud Applications

Data Preparation Methods

Methods of Data Preparation Description
Data Science Workspace - Data Prep Model driven transformation, scripted transformation.
Documentation
External ETL Tool (Snaplogic, Mulesoft, Informatica, and so on) Perform complex transformations in ETL tooling and use standard Experience Platform Flow Service APIs or source connectors to ingest the resultant data.
Query Service - Data Prep Joins, Splits, Merge, Transform, Query, and Filter data into a new dataset. Using Create Table as Select (CTAS)
Documentation
XDM Mapper & Data Prep functions (Streaming and Batch) Map source attributes in CSV or JSON format into XDM attributes during Experience Platform ingestion.
Compute functions on data as it is ingested; that is, data formatting, splitting, concatenation, and so on.
Documentation

On this page

Adobe Maker Awards Banner

Time to shine!

Apply now for the 2021 Adobe Experience Maker Awards.

Apply now
Adobe Maker Awards Banner

Time to shine!

Apply now for the 2021 Adobe Experience Maker Awards.

Apply now