Monitor sources dataflows using the Flow Service API
- Topics:
- Sources
CREATED FOR:
- Developer
This tutorial covers the steps for monitoring flow run data for completeness, errors, and metrics using the Flow Service API.
Getting started
This tutorial also requires you to have a working understanding of the following components of Adobe Experience Platform:
- Sources: Experience Platform allows data to be ingested from various sources while providing you with the ability to structure, label, and enhance incoming data using Experience Platform services.
- Sandboxes: Experience Platform provides virtual sandboxes which partition a single Experience Platform instance into separate virtual environments to help develop and evolve digital experience applications.
Using Experience Platform APIs
For information on how to successfully make calls to Experience Platform APIs, see the guide on getting started with Experience Platform APIs.
Monitor dataflows
To see the status of your dataflow, make a GET request to the Flow Service API, while providing your dataflow’s corresponding flow ID.
API format
GET /runs?property=flowId=={FLOW_ID}
Parameter | Description |
---|---|
{FLOW_ID} | The unique id value for the dataflow you want to monitor. |
Request
The following request retrieves the specifications for an existing dataflow.
curl -X GET \
'https://platform.adobe.io/data/foundation/flowservice/runs?property=flowId==c9cef9cb-c934-4467-8ef9-cbc934546741' \
-H 'Authorization: Bearer {ACCESS_TOKEN}' \
-H 'x-api-key: {API_KEY}' \
-H 'x-gw-ims-org-id: {ORG_ID}' \
-H 'x-sandbox-name: {SANDBOX_NAME}'
Response
A successful response returns details regarding your flow run, including information about its creation date, source and target connections, as well as the flow run’s unique identifier (id
).
{
"items": [
{
"createdAt": 1596656079576,
"updatedAt": 1596656113526,
"createdBy": "{CREATED_BY}",
"updatedBy": "{UPDATED_BY}",
"createdClient": "{CREATED_CLIENT}",
"updatedClient": "{UPDATED_CLIENT}",
"sandboxId": "1bd86660-c5da-11e9-93d4-6d5fc3a66a8e",
"sandboxName": "prod",
"id": "9830305a-985f-47d0-b030-5a985fd7d004",
"flowId": "c9cef9cb-c934-4467-8ef9-cbc934546741",
"etag": "\"b8003af1-0000-0200-0000-5f2b09f10000\"",
"metrics": {
"durationSummary": {
"startedAtUTC": 1596656058198,
"completedAtUTC": 1596656113306
},
"sizeSummary": {
"inputBytes": 24012,
"outputBytes": 17128
},
"recordSummary": {
"inputRecordCount": 100,
"outputRecordCount": 99,
"failedRecordCount": 1
},
"fileSummary": {
"inputFileCount": 1,
"outputFileCount": 1,
"activityRefs": [
"promotionActivity"
]
},
"statusSummary": {
"status": "success",
"errors": [
{
"code": "CONNECTOR-2001-500",
"message": "Error occurred at promotion activity."
}
],
"activityRefs": [
"promotionActivity"
]
}
},
"activities": [
{
"id": "copyActivity",
"updatedAtUTC": 1596656095088,
"durationSummary": {
"startedAtUTC": 1596656058198,
"completedAtUTC": 1596656089650,
"extensions": {
"windowStart": 1596653708000,
"windowEnd": 1596655508000
}
},
"sizeSummary": {
"inputBytes": 24012,
"outputBytes": 24012
},
"recordSummary": {},
"fileSummary": {
"inputFileCount": 1,
"outputFileCount": 1
},
"statusSummary": {
"status": "success",
"extensions": {
"type": "one-time"
}
},
"sourceInfo": [
{
"id": "c0e18602-f9ea-44f9-a186-02f9ea64f9ac",
"type": "SourceConnection",
"reference": {
"type": "AdfRunId",
"ids": [
"8a8eb0cc-e283-4605-ac70-65a5adb1baef"
]
}
}
]
},
{
"id": "promotionActivity",
"updatedAtUTC": 1596656113485,
"durationSummary": {
"startedAtUTC": 1596656095333,
"completedAtUTC": 1596656113306
},
"sizeSummary": {
"inputBytes": 24012,
"outputBytes": 17128
},
"recordSummary": {
"inputRecordCount": 100,
"outputRecordCount": 99,
"failedRecordCount": 1
},
"fileSummary": {
"inputFileCount": 2,
"outputFileCount": 1,
"extensions": {
"manifest": {
"fileInfo": "https://platform.adobe.io/data/foundation/export/batches/01EF01X41KJD82Y9ZX6ET54PCZ/meta?path=input_files"
}
}
},
"statusSummary": {
"status": "success",
"errors": [
{
"code": "CONNECTOR-2001-500",
"message": "Error occurred at promotion activity."
}
],
"extensions": {
"manifest": {
"failedRecords": "https://platform.adobe.io/data/foundation/export/batches/01EF01X41KJD82Y9ZX6ET54PCZ/meta?path=row_errors",
"sampleErrors": "https://platform.adobe.io/data/foundation/export/batches/01EF01X41KJD82Y9ZX6ET54PCZ/meta?path=row_error_samples.json"
},
"errors": [
{
"code": "INGEST-1212-400",
"message": "Encountered 1 errors in the data. Successfully ingested 99 rows. Review the associated diagnostic files for additional details."
},
{
"code": "MAPPER-3700-400",
"recordCount": 1,
"message": "Mapper Transform Error"
}
]
}
},
"targetInfo": [
{
"id": "47166b83-01c7-4b65-966b-8301c70b6562",
"type": "TargetConnection",
"reference": {
"type": "Batch",
"ids": [
"01EF01X41KJD82Y9ZX6ET54PCZ"
]
}
}
]
}
]
}
],
"_links": {}
}
Property | Description |
---|---|
items | Contains a single payload of metadata associated with your specific flow run. |
metrics | Defines characteristics of the data in the flow run. |
activities | Defines how the data is transformed. |
durationSummary | Defines the start and end time of the flow run. |
sizeSummary | Defines the volume of the data in bytes. |
recordSummary | Defines the record count of the data. |
fileSummary | Defines the file count of the data. |
statusSummary | Defines whether the flow run is a success or a failure. |
Next steps
By following this tutorial, you have retrieved metrics and error information on your dataflow using the Flow Service API. You can now continue to monitor your dataflow, depending on your ingestion schedule, to track its status and ingestion rates. For information on how to perform the same tasks using the user interface, see the tutorial on monitoring dataflows using the user interface
Experience Platform
- Sources overview
- Available source connectors
- Adobe applications
- Advertising
- Analytics
- Cloud storage
- Amazon Kinesis connector
- Amazon S3 connector
- Apache HDFS connector
- Azure Data Lake Storage Gen2 connector
- Azure Blob connector
- Azure Event Hubs connector
- Azure File Storage connector
- Data Landing Zone
- FTP connector
- Google Cloud Storage connector
- Google PubSub
- Oracle Object Storage
- SFTP connector
- Amazon S3 and Azure Blob connector
- Consent & Preferences
- CRM
- Customer success
- Databases
- Amazon Redshift connector
- Apache Hive on Azure HDInsights connector
- Apache Spark on Azure HDInsights connector
- Azure Databricks connector
- Azure Data Explorer connector
- Azure Synapse Analytics connector
- Azure Table Storage connector
- Google BigQuery connector
- GreenPlum connector
- HP Vertica connector
- IBM DB2 connector
- MariaDB connector
- Microsoft SQL Server connector
- MySQL connector
- Oracle connector
- PostgreSQL connector
- Snowflake Streaming connector
- Snowflake connector
- Teradata Vantage connector
- Data & identity partner
- eCommerce
- Local system
- Marketing automation
- Payments
- Protocols
- Streaming
- API tutorials
- Create a base connection
- Explore data
- Collect data
- On-demand ingestion
- Filter data at the source level
- Monitor dataflows
- Update accounts
- Update dataflows
- Retry failed dataflow runs
- Delete accounts
- Delete dataflows
- Ingest encrypted data
- Save a dataflow as a draft
- Apply access labels to a dataflow
- UI tutorials
- Create a source connection
- Adobe applications
- Advertising
- Analytics
- Cloud storage
- Consent & Preferences
- CRM
- Customer Success
- Databases
- Amazon Redshift
- Apache Hive on Azure HDInsights
- Apache Spark on Azure HDInsights
- Azure Databricks
- Azure Data Explorer
- Azure Synapse Analytics
- Azure Table Storage
- Google Big Query
- GreenPlum
- HP Vertica
- IBM DB2
- MariaDB
- Microsoft SQL Server
- MySQL
- Oracle
- PostgreSQL
- Snowflake
- Snowflake Streaming
- Teradata Vantage
- Data & identity partner
- eCommerce
- Local system
- Marketing automation
- Payments
- Protocols
- Streaming
- Configure a dataflow
- Advertising connection dataflow
- Analytics connection dataflow
- Batch cloud storage connection dataflow
- Streaming cloud storage connection dataflow
- Consent & Preferences connection dataflow
- CRM connection dataflow
- Customer success connection dataflow
- Database connection dataflow
- Ecommerce connection dataflow
- Marketing automation connection dataflow
- Payment connection dataflow
- Protocol connection dataflow
- Create a sources dataflow using templates in the UI
- Ingest encrypted data
- On-demand ingestion
- Monitor batch dataflows
- Monitor streaming dataflows
- Update accounts
- Update dataflows
- Delete accounts
- Delete dataflows
- Subscribe to sources alerts
- Save a dataflow as a draft
- Apply access labels to a dataflow
- Create a source connection
- Self-Serve Sources (Batch SDK)
- Overview
- Configure your connection specification
- Self-Serve Sources (Batch SDK) API guide
- Documentation guide
- Streaming SDK
- Get started with Self-Serve Sources (Streaming SDK)
- Create a connection specification for a streaming source
- Update a connection specification for a streaming source
- Update the streaming flow specification
- Test and submit your connection specification for verification
- Document your source (Streaming SDK)
- Documentation self-service API streaming template
- Documentation self-service UI streaming template
- Error messages
- Flow run notifications
- IP address allow list
- Frequently asked questions
- API reference
- Experience Platform release notes