Partial batch ingestion

Partial batch ingestion is the ability to ingest data containing errors, up to a certain threshold. With this capability, users can successfully ingest all their correct data into Adobe Experience Platform while all their incorrect data is batched separately, along with details as to why it is invalid.

This document provides a tutorial for managing partial batch ingestion.

Getting started

This tutorial requires a working knowledge of the various Adobe Experience Platform services involved with partial batch ingestion. Before beginning this tutorial, please review the documentation for the following services:

The following sections provide additional information that you will need to know in order to successfully make calls to Platform APIs.

Reading sample API calls

This guide provides example API calls to demonstrate how to format your requests. These include paths, required headers, and properly formatted request payloads. Sample JSON returned in API responses is also provided. For information on the conventions used in documentation for sample API calls, see the section on how to read example API calls in the Experience Platform troubleshooting guide.

Gather values for required headers

In order to make calls to Platform APIs, you must first complete the authentication tutorial. Completing the authentication tutorial provides the values for each of the required headers in all Experience Platform API calls, as shown below:

All resources in Experience Platform are isolated to specific virtual sandboxes. All requests to Platform APIs require a header that specifies the name of the sandbox the operation will take place in:

Enable a batch for partial batch ingestion in the API enable-api

You can create a new batch with partial ingestion enabled.

To create a new batch, follow the steps in the batch ingestion developer guide. Once you reach the Create batch step, add the following field within the request body:

{
    "enableErrorDiagnostics": true,
    "partialIngestionPercent": 5
}

Enable a batch for partial batch ingestion in the UI enable-ui

To enable a batch for partial ingestion through the Platform UI, you can create a new batch through source connections, create a new batch in an existing dataset, or create a new batch through the “Map CSV to XDM flow”.

Create a new source connection new-source

To create a new source connection, follow the listed steps in the Sources overview. Once you reach the Dataflow detail step, take note of the Partial ingestion and Error diagnostics fields.

The Partial ingestion toggle allows you to enable or disable the use of partial batch ingestion.

The Error diagnostics toggle only appears when the Partial ingestion toggle is off. This feature allows Platform to generate detailed error messages about your ingested batches. If the Partial ingestion toggle is turned on, enhanced error diagnostics are automatically enforced.

The Error threshold allows you to set the percentage of acceptable errors before the entire batch will fail. By default, this value is set to 5%.

Use an existing dataset existing-dataset

To use an existing dataset, start by selecting a dataset. The sidebar on the right populates with information about the dataset.

The Partial ingestion toggle allows you to enable or disable the use of partial batch ingestion.

The Error diagnostics toggle only appears when the Partial ingestion toggle is off. This feature allows Platform to generate detailed error messages about your ingested batches. If the Partial ingestion toggle is turned on, enhanced error diagnostics are automatically enforced.

The Error threshold allows you to set the percentage of acceptable errors before the entire batch will fail. By default, this value is set to 5%.

Now, you can upload data using the Add data button, and it will be ingested using partial ingestion.

Use the “Map CSV to XDM schema” flow map-flow

To use the “Map CSV to XDM schema” flow, follow the listed steps in the Map a CSV file tutorial. Once you reach the Add data step, take note of the Partial ingestion and Error diagnostics fields.

The Partial ingestion toggle allows you to enable or disable the use of partial batch ingestion.

The Error diagnostics toggle only appears when the Partial ingestion toggle is off. This feature allows Platform to generate detailed error messages about your ingested batches. If the Partial ingestion toggle is turned on, enhanced error diagnostics are automatically enforced.

Error threshold allows you to set the percentage of acceptable errors before the entire batch will fail. By default, this value is set to 5%.

Next steps next-steps

This tutorial covered how to create or modify a dataset to enable partial batch ingestion. For more information on batch ingestion, please read the batch ingestion developer guide.

For information on monitoring partial ingestion errors, please read the batch ingestion error diagnostics guide.