Create a flow run for on-demand ingestion using the Flow Service API
Flow runs represent an instance of flow execution. For example, if a flow is scheduled to run hourly at 9:00 AM, 10:00 AM, and 11:00 AM, then you would have three instances of a flow run. Flow runs are specific to your particular organization.
On-demand ingestion provides you with the capability to create a flow run against a given dataflow. This allows your users to create a flow run, based on given parameters and create an ingestion cycle, without service tokens. Support for on-demand ingestion is available only for batch sources.
This tutorial covers the steps on how to use on-demand ingestion and create a flow run using the Flow Service API.
Getting started
This tutorial requires you to have a working understanding of the following components of Adobe Experience Platform:
- Sources: Experience Platform allows data to be ingested from various sources while providing you with the ability to structure, label, and enhance incoming data using Platform services.
- Sandboxes: Experience Platform provides virtual sandboxes which partition a single Platform instance into separate virtual environments to help develop and evolve digital experience applications.
Using Platform APIs
For information on how to successfully make calls to Platform APIs, see the guide on getting started with Platform APIs.
Create a flow run for a table-based source
To create a flow for a table-based source, make a POST request to the Flow Service API while providing the ID of the flow you want to create the run against, as well as values for start time, end time, and delta column.
API format
POST /runs/
Request
The following request creates a flow run for flow ID 3abea21c-7e36-4be1-bec1-d3bad0e3e0de
.
deltaColumn
when creating your first flow run. After that, deltaColumn
will be patched as part of copy
transformation in the flow and will be treated as the source of truth. Any attempts to change the deltaColumn
value through the flow run parameters will result in an error.curl -X POST \
'https://platform.adobe.io/data/foundation/flowservice/runs' \
-H 'Authorization: Bearer {ACCESS_TOKEN}' \
-H 'x-api-key: {API_KEY}' \
-H 'x-gw-ims-org-id: {ORG_ID}' \
-H 'x-sandbox-name: {SANDBOX_NAME}' \
-H 'Content-Type: application/json'
-d '{
"flowId": "3abea21c-7e36-4be1-bec1-d3bad0e3e0de",
"params": {
"startTime": "1663735590",
"windowStartTime": "1651584991",
"windowEndTime": "16515859567",
"deltaColumn": {
"name": "DOB"
}
}
}'
flowId
params.startTime
params.windowStartTime
params.windowEndTime
params.deltaColumn
deltaColumn
is only needed when creating your firs flow run.params.deltaColumn.name
Response
A successful response returns the details of the newly created flow run, including its unique run id
.
{
"items": [
{
"id": "3fb0418e-1804-45d6-8d56-dd51f05c0baf",
"etag": "\"1100c53e-0000-0200-0000-627138980000\""
}
]
}
id
etag
Create a flow run for a file-based source
To create a flow for a file-based source, make a POST request to the Flow Service API while providing the ID of the flow you want to create the run against and values for start time and end time.
API format
POST /runs/
Request
The following request creates a flow run for flow ID 3abea21c-7e36-4be1-bec1-d3bad0e3e0de
.
curl -X POST \
'https://platform.adobe.io/data/foundation/flowservice/runs' \
-H 'Authorization: Bearer {ACCESS_TOKEN}' \
-H 'x-api-key: {API_KEY}' \
-H 'x-gw-ims-org-id: {ORG_ID}' \
-H 'x-sandbox-name: {SANDBOX_NAME}' \
-H 'Content-Type: application/json'
-d '{
"flowId": "3abea21c-7e36-4be1-bec1-d3bad0e3e0de",
"params": {
"startTime": "1663735590",
"windowStartTime": "1651584991",
"windowEndTime": "16515859567"
}
}'
flowId
params.startTime
params.windowStartTime
params.windowEndTime
Response
A successful response returns the details of the newly created flow run, including its unique run id
.
{
"items": [
{
"id": "3fb0418e-1804-45d6-8d56-dd51f05c0baf",
"etag": "\"1100c53e-0000-0200-0000-627138980000\""
}
]
}
id
etag
Monitor your flow runs
Once your flow run has been created, you can monitor the data that is being ingested through it to see information on flow runs, completion status, and errors. To monitor your flow runs using the API, see the tutorial on monitoring dataflows in the API. To monitor your flow runs using Platform UI, see the guide on monitoring sources dataflows using the monitoring dashboard.