Create a dataflow for marketing automation sources using the Flow Service API
This tutorial covers the steps for retrieving data from a marketing automation source and bringing them to Experience Platform using Flow Service API.
NOTE- In order to create a dataflow, you must already have a valid base connection ID with a marketing automation source. If you do not have this ID, then see the sources overview for a list of marketing automation sources that you can create a base connection with.
- For Experience Platform to ingest data, timezones for all table-based batch sources must be configured to UTC.
Getting started
This tutorial also requires you to have a working understanding of the following components of Adobe Experience Platform:
-
Experience Data Model (XDM) System: The standardized framework by which Experience Platform organizes customer experience data.
- Basics of schema composition: Learn about the basic building blocks of XDM schemas, including key principles and best practices in schema composition.
- Schema Registry developer guide: Includes important information that you need to know in order to successfully perform calls to the Schema Registry API. This includes your
{TENANT_ID}, the concept of “containers”, and the required headers for making requests (with special attention to the Accept header and its possible values).
-
Catalog Service: Catalog is the system of record for data location and lineage within Experience Platform.
-
Batch ingestion: The Batch Ingestion API allows you to ingest data into Experience Platform as batch files.
-
Sandboxes: Experience Platform provides virtual sandboxes which partition a single Experience Platform instance into separate virtual environments to help develop and evolve digital experience applications.
Using Experience Platform APIs
For information on how to successfully make calls to Experience Platform APIs, see the guide on getting started with Experience Platform APIs.
Create a source connection
You can create a source connection by making a POST request to the Flow Service API. A source connection consists of a connection ID, a path to the source data file, and a connection spec ID.
To create a source connection, you must also define an enum value for the data format attribute.
Use the following the enum values for file-based connectors:
| Data format | Enum value |
|---|---|
| Delimited | delimited |
| JSON | json |
| Parquet | parquet |
For all table-based connectors, set the value to tabular.
API format
POST /sourceConnections
Request
curl -X POST \
'https://platform.adobe.io/data/foundation/flowservice/sourceConnections' \
-H 'Authorization: Bearer {ACCESS_TOKEN}' \
-H 'x-api-key: {API_KEY}' \
-H 'x-gw-ims-org-id: {ORG_ID}' \
-H 'x-sandbox-name: {SANDBOX_NAME}' \
-H 'Content-Type: application/json' \
-d '{
"name": "HubSpot source connection",
"baseConnectionId": "c6d4ee17-6752-4e83-94ee-1767522e83fa",
"description": "HubSpot source connection",
"data": {
"format": "tabular",
},
"params": {
"path": "Hubspot.Contacts"
},
"connectionSpec": {
"id": "cc6a4487-9e91-433e-a3a3-9cf6626c1806",
"version": "1.0"
}
}'
| Property | Description |
|---|---|
baseConnectionId | The unique connection ID of the third-party marketing automation system you are accessing. |
params.path | The path of the source file you are accessing. |
connectionSpec.id | The connection specification ID of your marketing automation system. |
Response
A successful response returns the unique identifier (id) of the newly created source connection. Store this value as it is required in later steps for creating a target connection.
{
"id": "f44dbef2-a4f0-4978-8dbe-f2a4f0e978cf",
"etag": "\"5f00fba7-0000-0200-0000-5ed560520000\""
}