Create a flow run for on-demand ingestion using the Flow Service API

Flow runs represent an instance of flow execution. For example, if a flow is scheduled to run hourly at 9:00 AM, 10:00 AM, and 11:00 AM, then you would have three instances of a flow run. Flow runs are specific to your particular organization.

On-demand ingestion provides you with the capability to create a flow run against a given dataflow. This allows your users to create a flow run, based on given parameters and create an ingestion cycle, without service tokens. Support for on-demand ingestion is available only for batch sources.

This tutorial covers the steps on how to use on-demand ingestion and create a flow run using the Flow Service API.

Getting started

This tutorial requires you to have a working understanding of the following components of Adobe Experience Platform:

Using Platform APIs

For information on how to successfully make calls to Platform APIs, see the guide on getting started with Platform APIs.

Create a flow run for a table-based source

To create a flow for a table-based source, make a POST request to the Flow Service API while providing the ID of the flow you want to create the run against, as well as values for start time, end time, and delta column.

API format

POST /runs/

Request

The following request creates a flow run for flow ID 3abea21c-7e36-4be1-bec1-d3bad0e3e0de.

curl -X POST \
  'https://platform.adobe.io/data/foundation/flowservice/runs' \
  -H 'Authorization: Bearer {ACCESS_TOKEN}' \
  -H 'x-api-key: {API_KEY}' \
  -H 'x-gw-ims-org-id: {ORG_ID}' \
  -H 'x-sandbox-name: {SANDBOX_NAME}' \
  -H 'Content-Type: application/json'
  -d '{
      "flowId": "3abea21c-7e36-4be1-bec1-d3bad0e3e0de",
      "params": {
          "startTime": "1663735590",
          "windowStartTime": "1651584991",
          "windowEndTime": "16515859567",
          "deltaColumn": {
              "name": "DOB"
          }
      }
  }'

Response

A successful response returns the details of the newly created flow run, including its unique run id.

{
    "items": [
        {
            "id": "3fb0418e-1804-45d6-8d56-dd51f05c0baf",
            "etag": "\"1100c53e-0000-0200-0000-627138980000\""
        }
    ]
}

Create a flow run for a file-based source

To create a flow for a file-based source, make a POST request to the Flow Service API while providing the ID of the flow you want to create the run against and values for start time and end time.

API format

POST /runs/

Request

The following request creates a flow run for flow ID 3abea21c-7e36-4be1-bec1-d3bad0e3e0de.

curl -X POST \
  'https://platform.adobe.io/data/foundation/flowservice/runs' \
  -H 'Authorization: Bearer {ACCESS_TOKEN}' \
  -H 'x-api-key: {API_KEY}' \
  -H 'x-gw-ims-org-id: {ORG_ID}' \
  -H 'x-sandbox-name: {SANDBOX_NAME}' \
  -H 'Content-Type: application/json'
  -d '{
      "flowId": "3abea21c-7e36-4be1-bec1-d3bad0e3e0de",
      "params": {
          "startTime": "1663735590",
          "windowStartTime": "1651584991",
          "windowEndTime": "16515859567"
      }
  }'

Response

A successful response returns the details of the newly created flow run, including its unique run id.

{
    "items": [
        {
            "id": "3fb0418e-1804-45d6-8d56-dd51f05c0baf",
            "etag": "\"1100c53e-0000-0200-0000-627138980000\""
        }
    ]
}

Monitor your flow runs

Once your flow run has been created, you can monitor the data that is being ingested through it to see information on flow runs, completion status, and errors. To monitor your flow runs using the API, see the tutorial on monitoring dataflows in the API. To monitor your flow runs using Platform UI, see the guide on monitoring sources dataflows using the monitoring dashboard.