Create a data feed
When creating a data feed, you provide Adobe with:
-
The information about the destination where you want raw data files to be sent
-
The data you want to include in each file
-
The frequency with which data is sent (including the processing delay to capture late-arriving events)
Before you create a data feed, it’s important to have a basic understanding of data feeds and to ensure that you meet all prerequisites. For more information, see Data feeds overview.
Create and configure a data feed create-and-configure-data-feed
-
Log in to experiencecloud.adobe.com using your Adobe ID credentials.
-
Select Customer Journey Analytics from the app switcher
at the top right of the interface. -
In the top navigation bar, go to Components > Exports.
-
Select the Data feeds tab.
-
Select Create in the upper-right corner of the screen.
Or, if no data feeds have previously been created, select Create data feed within the empty table.
A page displays with the following tabs: Details, Data structure, and Delivery.
-
On the Details tab, complete the following fields:
table 0-row-2 1-row-2 2-row-2 3-row-2 4-row-2 Field Function Name The name of the data feed. Names must be unique within the selected data view, and can be up to 255 characters in length. Tags Apply any tags to the data feed for easier categorization. Description Specify a description for the data feed (up to 500 characters). The description you add is visible when editing the data feed. Data view Select the data view that contains the data that you want to export.
Consider the following when selecting a data view:
- If multiple data feeds are created for the same data view, each data feed must have different column definitions.
- The list of available columns depends on the login company where the selected data view belongs. If you change the data view, the list of available columns can change.
-
Select Next.
-
On the Data structure tab, make sure the correct data view is selected in the Data view field.
-
In the Segments drop-down menu, search for and select any segments to filter the data included in your feed.
When you apply multiple segments, they are joined together with an AND operator. To join segments with an OR operator, you must first create a new segment in the segment builder, then apply the new segment to the data feed.
Segments you apply here are in addition to any segments that might already be applied in your data view.
-
(Optional) In the left rail, use the search field to locate specific components. Or, select the Sort icon
to apply any of the following sort options:table 0-row-2 1-row-2 2-row-2 3-row-2 Option Function Recommended Sorts components with those that are recommended at the top of the list. Components that are used most frequently and most recently by you or by others in your organization are shown higher in the list. Alphabetical Sorts components alphabetically. Categorical Sorts components similar to Recommended, except that calculated metrics and standard metrics are grouped separately instead of being mixed together. -
Add components to the data feed configuration. The left rail shows only components that are valid for data feeds.
- Drag-and-drop: Drag components from the left rail to the canvas. Hold Shift, or hold Command (macOS) or Ctrl (Windows) to select and drag multiple components at once.
- Plus button: Select the Plus
icon next to any component in the left rail to add it to the canvas. - Show all: Select Show all at the bottom of the component list to open a dialog showing all available components. Select the checkbox next to each component you want to add, then select Add selected. When a search term or filter tag is active in the left rail, an Add all button also appears, letting you add all filtered results at once.
Consider the following when adding fields:
-
Some components are required, unsupported, or have restrictions in data feeds. For details, see Component availability in data feeds.
-
When you add a component that belongs to an XDM array field (for example, an Adobe Journey Optimizer proposition field) or a map field, a dialog prompts you to add any other components from the same sub-event. In the data feed output, all of these components appear in a single column. For more information, see Sub-event components in data feeds
-
(Optional) Reorder components on the canvas by dragging them. The order you define is preserved as the column order in the exported data feed file.
-
(Optional) Resize columns on the canvas by dragging the column border.
Column widths are saved in a cookie and persist the next time you return to this data feed on the same browser.
-
(Optional) Change the output ID of a component. The output ID is the column name in the data feed output.
By default, the output ID is the component ID in snake_case. Changing the output ID affects only this data feed. It does not change the component ID in the data view. The schema preview shows both the component ID and the output ID.
-
Hover over a component on the canvas, then select the information icon
. -
Select the edit icon
next to the ID, then specify a new output ID.
Output IDs must:
- Be unique within the data feed.
- Contain only letters, numbers, and underscores (
_). Spaces, periods (.), hyphens (-), forward slashes (/), and other special characters are not allowed. - Begin with a letter or an underscore.
- Be 255 characters or fewer.
Output IDs cannot begin with
_cja, which is reserved for Adobe use. -
-
(Optional) Use the Feed summary and Schema preview panels on the right side of the page to review your data structure before proceeding:
- The Feed summary shows a live count of the total components, columns, dimensions, and metrics that you added.
- The Schema preview shows a JSON representation of the data feed schema that updates as you add or reorder components.
- The Example rows button opens a dialog that shows example output rows so you can verify that the structure looks correct. This dialog shows example data only and does not reflect your actual data.
-
On the Delivery tab, in the Schedule section, choose the type of feed you want to create (live or backfill), then specify the reporting window, frequency, and other configuration options:
table 0-row-2 1-row-2 2-row-2 3-row-2 4-row-2 5-row-2 6-row-2 7-row-2 8-row-2 9-row-2 Field Function Feed type Select the type of feed you want to create:
- Live feed: Exports current and future data.
- Backfill feed: Exports historical data.
Start date The date when the data feed begins. For live feeds, this must be today or a future date. For backfill feeds, this must be a past date within the data view’s data retention window. The start date is based on the data view’s time zone. Expiration date
Available only for live feedsThe date when the data feed expires and no longer runs. The date is based on the data view’s time zone. End date
Available only for backfill feedsThe date when the data feed ends. The end date cannot be in the future. The date is based on the data view’s time zone. Frequency
Available only for live feedsSelect how often the data feed should be sent. Events with timestamps that fall within the frequency window are included in the data feed delivery. The Lookback date range and Processing delay fields can also affect which events are included in the data for the delivery frequency that you choose.
Select to include either one hour’s worth of data or one day’s worth of data.
- Daily: Feeds contain a full day’s worth of data, from midnight to midnight in the data view’s time zone.
- Hourly: Feeds contain a single hour’s worth of data.
Granularity
Available only for backfill feedsThe time interval used to divide historical data into chunks. Each chunk contains a full day’s worth of data, from midnight to midnight in the data view’s time zone.
Granularity determines how the data is grouped, not how often it is delivered. Backfill data is delivered as quickly as possible, not once per day.
This field is always set to Daily and cannot be modified.
Lookback date range Controls how far back Customer Journey Analytics looks when processing the data feed delivery. The default is 30 days.
The frequency window (hour or day) determines which events are included in the data feed, while the lookback date range provides the needed historical context to classify those events correctly.
Segment qualification, dimension persistence, session calculation, and derived field transformations can all affect the events that are included.
Before configuring this option, see the details and examples described in the section below, Understand the lookback date range.
Processing delay The amount of time that Customer Journey Analytics waits before processing a data feed file. Any late-arriving events that come in during the processing delay are included in the data feed.
Choose a delay that is long enough for the slowest data in your connection to finish processing. If the delay is too short, data that is still processing is not included in the data feed file.
The minimum processing delay is 2 hours. The default is based on the datasets in your connection. For example, if stitching is enabled, the default is 6 hours. Some types of data require a longer delay. The delay you choose depends on the types of data in your connection, such as streaming, batch, stitched, lookup, or profile data.
Before configuring this option, see the details and examples described in the section below, Understand the processing delay.
Compression format Select the compression format for the Parquet output files delivered to your cloud destination. Choose from the following formats:
- Snappy: Fast compression and decompression with moderate file sizes. Widely supported by modern data platforms such as BigQuery, Snowflake, and Apache Spark.
- GZip: Broadly compatible, including with tools that do not natively support Snappy. Recommended if your downstream pipeline requires a widely recognized compression standard.
- Z Standard (Zstd): High compression efficiency with fast decompression. Suitable if minimizing file size is a priority and your tools support Zstd.
-
On the Delivery tab, in the Destination section, configure the destination where you want the data to be sent.
note NOTE Consider the following when configuring a report destination: -
Any cloud accounts that you previously configured are available to use for data feeds. You can configure cloud accounts from the Locations manager, in Components > Exports > Location accounts.
-
Cloud accounts are associated with your Customer Journey Analytics user account. Other users cannot use or view cloud accounts that you configure unless you make them available to all users in your organization.
-
You can edit any locations that you create from the Locations manager in Components > Exports > Locations.
Complete the following fields:
table 0-row-2 1-row-2 2-row-2 3-row-2 4-row-2 5-row-2 6-row-2 7-row-2 Field Function View destinations for all users If you are a system administrator, you can enable this option to view destinations created by all users in your organization. When this option is disabled, only destinations you created are displayed. Account Do either of the following:
-
Use an existing account: Select the drop-down menu next to the Account field. Or, begin typing the account name, then select it from the drop-down menu.
Accounts are available to you only if you configured them or if they are shared with an organization you are a part of.
-
Create a new account: Select Add account within the Account drop-down menu. For information about how to configure the account, see Configure cloud export accounts.
Location Do either of the following:
- Use an existing location: Select the drop-down menu next to the Location field. Or, begin typing the location name, then select it from the drop-down menu.
- Create a new location: Select Add location within the Location drop-down menu. For information about how to configure the location, see Configure cloud export locations.
Override default path or file name Enable this option to override the default path or file name that is set for the selected location. The override applies only to this data feed and determines where data feed files are sent.
When this option is enabled, the Path or file name field displays.
Path or file name
Available only when Override default path or file name is enabledSpecify a path or file name (or both) to determine where files are sent within the selected location. To save files to the top-level folder of the location, specify
/.For live feeds, you can use any of the following variables to make the path or file name dynamic:
- {yyyy}: 4-digit calendar year (Case sensitive)
- {yy}: 2-digit calendar year (Case sensitive)
- {MM}: 2-digit month (Case sensitive)
- {dd}: 2-digit day (Case sensitive)
- {HH}: 2-digit hour (Case sensitive)
- {mm}: 2-digit minutes (Case sensitive)
- {ss}: 2-digit seconds (Case sensitive)
- {fff}: 3-digit nanoseconds (Case sensitive)
- {instance_id}: Request (instance) UUID
- {export_id}: Export (schedule) UUID
- {idx}: Index start from 0 (incremented for each file)
- {total}: Total file number for whole transfer job
- {completion_millis}: Time of transfer in milliseconds
For example, if you specify
${yyyy}/${MM}/${dd}/my-feed-${instance_id}-${idx}, a live feed that is delivered on January 15, 2027 would have the following file path and name:[prefix_folder_name]/2027/01/15/my-feed-[UUID]-1.parquetNotify of issues, when complete, and when expiring Specify one or more email addresses where a notification should be delivered when the data feed completes, is expiring, or encounters issues. Separate multiple email addresses with a comma. Enable manifest Choose whether to include a manifest file with each data feed delivery. The manifest file contains information for each file included in the data feed. -
-
Select Save.
Understand the lookback date range data-feed-lookback-date-range
The lookback date range controls how far back Customer Journey Analytics looks when processing each data feed delivery.
Events must still have timestamps that fall within the frequency window (hour or day) to be included in the delivery, but the data that falls within the lookback date range provides the needed historical context to classify those events correctly.
When configuring this option, consider the following important concepts:
- A longer lookback date range typically results in more accurate data; a shorter range results in better delivery performance.
- The lookback date range, together with the frequency window, function similarly to the Analysis Workspace reporting date range. However, there are key differences. These differences can result in data discrepancies between Workspace reports and data feed deliveries.
Segment qualification, session calculation, dimension persistence, and derived field transformations are each considered when processing data within the lookback date range:
Segment qualification
When a segment is applied to your data feed definition, data within the lookback date range determines which events, sessions or people qualify for the segment. The segment’s container setting determines the scope. (Possible containers are: Person, Session, or Event. B2B includes the following additional containers: Global account, Account, Opportunity, Buying group.)
Example:
Suppose you want to create a data feed to understand the behavior of users who are part of a specific marketing campaign, Campaign B.
To accomplish this, you apply a segment to the data feed called Users in Campaign B, indicating that only those events tied to users in this segment should be included in the data feed.
In this case, users are included in the data feed only if they meet both of the following conditions:
-
The user had an event with a timestamp that is within the data feed frequency window (the given hour or day of the data feed).
-
The user qualified for the Campaign B segment sometime within the lookback date range.
For a qualifying event that occurred 9 days ago, this means the user would be included in the data feed if the lookback date range were set to 30 days, but the user would not be included in the data feed if the lookback date range were set to 7 days.
Session calculation
Session boundaries are calculated using all events in the lookback date range, not just the events in the delivery window. A session that started before the delivery window is still recognized as the same session.
The Session ID is based on the person, the session start time, and the session settings in your data view. A session keeps the same Session ID across deliveries, so you can join events from a session that spans multiple hourly or daily deliveries.
Consider the following when working with sessions in data feeds:
- If a session started before the lookback date range, its earlier events are not available, so session values can differ from Analysis Workspace. For more information, see Understand data discrepancies between data feeds and Analysis Workspace.
- Changing the session settings in your data view changes Session IDs. Session IDs in later deliveries won’t match Session IDs in earlier deliveries.
Dimension persistence
When you set persistence on an individual dimension, you also set an expiration to determine how long the dimension item persists beyond the event it is set on.
The lookback date range affects dimension persistence when the expiration is set to either of the following options in the data view:
-
Person Reporting Window: The lookback date range becomes the new reporting window for each dimension in the data feed definition that uses Person Reporting Window as its expiration.
-
Custom Time: If the custom time that is selected extends beyond the lookback date range, the custom time is ignored, and the lookback date range is used for dimension expiration for each dimension in the data feed definition that uses Custom Time as its expiration. Values that occurred before the lookback date range are not considered.
For more information about setting persistence on dimensions within the data view, see Persistence component settings.
To get the most accurate data, consider setting the lookback date range to a value equal to or greater than the persistence set on dimensions in your data. However, keep in mind that a shorter lookback date range results in better performance for data feed deliveries.
Example:
Suppose that in your data feed you want to know which marketing campaign users originally saw before coming to your site.
To accomplish this, you set persistence on the Campaigns dimension with Original as the allocation model.
In this case, the original campaign is shown in the data feed output only if users meet both of the following conditions:
-
The user had an event with a timestamp that is within the data feed frequency window (the given hour or day of the data feed).
-
The user qualified for the original campaign sometime within the lookback date range.
If the user qualified for the original campaign 9 days ago, the original campaign is included in the data feed if the lookback date range is set to 30 days, but the original campaign is not included in the data feed if the lookback date range is set to 7 days.
Derived field transformations
Any derived field functions that reference containers use the lookback date range in data feed exports. What date capabilities exist in derived fields?
Understand the processing delay data-feed-processing-delay
How the processing delay works
The processing delay is the amount of time that Customer Journey Analytics waits before processing a data feed file. Any late-arriving events that come in during the processing delay are included in the data feed.
Processing delays are needed for various reasons, such as to account for pipeline latency, to give mobile implementations an opportunity for offline devices to come online and send data, or to accommodate your organization’s server-side processes in managing previously processed files.
The minimum processing delay is 2 hours, but some types of data require a longer delay.
Example:
Suppose an hourly data feed includes data from 1:00 PM to 2:00 PM and the processing delay is 2 hours. Processing for that data feed file begins at 4:00 PM and includes any data that arrived before processing begins.
Choose a processing delay based on your data
Different types of data take varying amounts of time to become available in Customer Journey Analytics. Data goes through two processing phases, and the time for each phase adds to the total.
Choose a processing delay that is long enough for the slowest data in your connection to complete both phases. If the delay is too short, data that is still processing is not included in the data feed file.
When you create a data feed, the processing delay defaults to a recommended value based on the datasets in your connection. For example, if stitching is enabled for the connection, the default is 6 hours. You can choose a shorter delay, down to the 2-hour minimum, but a warning appears because some data might not be included in the data feed.
Phase 1: Data arrives in the Experience Platform data lake
Arrival times vary based on the type of data you are collecting. Choose a delay that accommodates the type of data you are collecting.
-
Event datasets from the Edge Network or streaming ingestion: Data typically arrives in the data lake within 60 minutes (see Latencies).
-
Analytics source connector datasets: Data typically arrives in the data lake within 2.25 hours (see Latencies).
-
Datasets from other source connectors: Latency varies by source connector and by when batches are sent. Upstream processing in Experience Platform, such as Data Prep, can add more time.
-
Lookup datasets: The time for data to arrive in the data lake depends on how often data is uploaded. Lookup data is typically uploaded as a full copy of a database, in which only a small percentage of records have changed. Upload lookup data in smaller batches to shorten processing time.
Small uploads are usually processed within the minimum delay.
Large uploads (for example, a weekly upload of millions of records) are processed at a lower priority and can take 3 to 4 hours longer. In the case of large uploads, event data is not delayed, but lookup values might not reflect the newest updates.
-
Profile datasets: The time for data to arrive in the data lake depends on how often data is uploaded. Profile data is typically ingested in large batches, such as a daily snapshot of the full profile table. Upload profile data in smaller batches to shorten processing time.
Small uploads are usually processed within the minimum delay.
Large uploads (for example, a weekly upload of millions of records) are processed at a lower priority and can take 3 to 4 hours longer. In the case of large uploads, event data is not delayed, but profile values might not reflect the newest updates.
Phase 2: Data is ingested from the data lake into Customer Journey Analytics
Data ingestion times vary depending on whether the dataset has stitching enabled.
-
Non-stitched datasets: This can take up to 90 minutes (see Latencies).
-
Stitched datasets: Stitching can add up to 4 hours on top of the 90 minutes it takes for non-stitched datasets (see Latencies). If stitching is enabled for the connection, the default delay is 6 hours. Data that is updated by a stitching replay is generally not included in data feed files that were already processed.
Example:
If your connection includes multiple types of data, choose a delay that accommodates the slowest data. In the example below, that is approximately 8 hours.
Stitching can add up to 4 hours to ingestion into Customer Journey Analytics. To account for this, add 4 hours to the delay for any stitched data.
90 minutes
Without stitching
90 minutes + 4 hours for stitching
With stitching