[Beta]{class="badge informative"}
Update flow specifications using the Flow Service API
Once you have generated a new connection specification ID, you must add this ID to a flow specification in order to create a dataflow.
Flow specifications contain information that defines a flow, including the source and target connection IDs that it supports, transformation specifications that are needed to be applied to the data, and scheduling parameters required to generate a flow. You can edit flow specifications by using the /flowSpecs endpoint.
The following document provides steps on how to retrieve and update flow specifications using the Flow Service API for Self-Serve Sources (Streaming SDK).
Getting started
Before continuing, please review the getting started guide for links to related documentation, a guide to reading the sample API calls in this document, and important information regarding required headers that are needed to successfully make calls to any Experience Platform API.
Look up a flow specification lookup
Sources created with the generic-streaming template all use the GenericStreamingAEP flow specification. This flow specification can be retrieved by making a GET request to the /flowSpecs/ endpoint, and providing the flowSpec.id of e77fde5a-22a8-11ed-861d-0242ac120002.
API format
GET /flowSpecs/e77fde5a-22a8-11ed-861d-0242ac120002
Request
The following request retrieves the e77fde5a-22a8-11ed-861d-0242ac120002 flow specification.
curl -X GET \
'https://platform.adobe.io/data/foundation/flowservice/flowSpecs/e77fde5a-22a8-11ed-861d-0242ac120002' \
-H 'Authorization: Bearer {ACCESS_TOKEN}' \
-H 'Content-Type: application/json' \
-H 'x-api-key: {API_KEY}' \
-H 'x-gw-ims-org-id: {ORG_ID}' \
-H 'x-sandbox-name: {SANDBOX_NAME}'
Response
A successful response returns the details of the queried flow specification.
{
"items": [
{
"id": "e77fde5a-22a8-11ed-861d-0242ac120002",
"name": "GenericStreamingAEP",
"providerId": "521eee4d-8cbe-4906-bb48-fb6bd4450033",
"version": "1.0",
"sourceConnectionSpecIds": [
"e77fd9d2-22a8-11ed-861d-0242ac120002"
],
"targetConnectionSpecIds": [
"c604ff05-7f1a-43c0-8e18-33bf874cb11c"
],
"optionSpec": {
"name": "OptionSpec",
"spec": {
"$schema": "http://json-schema.org/draft-07/schema#",
"type": "object",
"properties": {
"errorDiagnosticsEnabled": {
"title": "Error diagnostics.",
"description": "Flag to enable detailed and sample error diagnostics summary.",
"type": "boolean",
"default": false
},
"partialIngestionPercent": {
"title": "Partial ingestion threshold.",
"description": "Percentage which defines the threshold of errors allowed before the run is marked as failed.",
"type": "number",
"exclusiveMinimum": 0
},
"inletUrl": {
"title": "Inlet Url.",
"description": "Inlet Url which defines the streaming endpoint.",
"type": "string"
}
}
}
},
"transformationSpecs": [
{
"name": "Mapping",
"spec": {
"$schema": "http://json-schema.org/draft-07/schema#",
"type": "object",
"description": "defines various params required for different mapping from source to target",
"properties": {
"mappingId": {
"type": "string"
},
"mappingVersion": {
"type": "string"
}
}
}
}
],
"attributes": {
"isSourceFlow": true,
"flacValidationSupported": true,
"frequency": "streaming",
"uiAttributes": {
"apiFeatures": {
"updateSupported": true,
"flowRunsSupported": true
}
}
},
"permissionsInfo": {
"manage": [
{
"@type": "lowLevel",
"name": "StreamingSource",
"permissions": [
"write"
]
}
],
"view": [
{
"@type": "lowLevel",
"name": "StreamingSource",
"permissions": [
"read"
]
}
]
}
}
]
}
Update a flow specification update
You can update the fields of a flow specification through a PUT operation. When updating a flow specification through a PUT request, the body must include all of the fields that would be required when creating a new flow specification in a POST request.
sourceConnectionSpecIds array of the flow specifications that correspond to your source. This ensures that your new source is supported by an existing flow specification, thus allowing you to complete the dataflow creation process with your new source.API format
PUT /flowSpecs/e77fde5a-22a8-11ed-861d-0242ac120002
Request
The following request updates the flow specification of e77fde5a-22a8-11ed-861d-0242ac120002 to include the connection specification ID bdb5b792-451b-42de-acf8-15f3195821de.
PUT -X GET \
'https://platform.adobe.io/data/foundation/flowservice/connectionSpecs/e77fde5a-22a8-11ed-861d-0242ac120002' \
-H 'Authorization: Bearer {ACCESS_TOKEN}' \
-H 'Content-Type: application/json' \
-H 'x-api-key: {API_KEY}' \
-H 'x-gw-ims-org-id: {ORG_ID}' \
-H 'x-sandbox-name: {SANDBOX_NAME}'
-d '{
"id": "e77fde5a-22a8-11ed-861d-0242ac120002",
"name": "GenericStreamingAEP",
"providerId": "521eee4d-8cbe-4906-bb48-fb6bd4450033",
"version": "1.0",
"sourceConnectionSpecIds": [
"e77fd9d2-22a8-11ed-861d-0242ac120002",
"bdb5b792-451b-42de-acf8-15f3195821de"
],
"targetConnectionSpecIds": [
"c604ff05-7f1a-43c0-8e18-33bf874cb11c"
],
"optionSpec": {
"name": "OptionSpec",
"spec": {
"$schema": "http://json-schema.org/draft-07/schema#",
"type": "object",
"properties": {
"errorDiagnosticsEnabled": {
"title": "Error diagnostics.",
"description": "Flag to enable detailed and sample error diagnostics summary.",
"type": "boolean",
"default": false
},
"partialIngestionPercent": {
"title": "Partial ingestion threshold.",
"description": "Percentage which defines the threshold of errors allowed before the run is marked as failed.",
"type": "number",
"exclusiveMinimum": 0
},
"inletUrl": {
"title": "Inlet Url.",
"description": "Inlet Url which defines the streaming endpoint.",
"type": "string"
}
}
}
},
"transformationSpecs": [
{
"name": "Mapping",
"spec": {
"$schema": "http://json-schema.org/draft-07/schema#",
"type": "object",
"description": "defines various params required for different mapping from source to target",
"properties": {
"mappingId": {
"type": "string"
},
"mappingVersion": {
"type": "string"
}
}
}
}
],
"attributes": {
"isSourceFlow": true,
"flacValidationSupported": true,
"frequency": "streaming",
"uiAttributes": {
"apiFeatures": {
"updateSupported": true,
"flowRunsSupported": true
}
}
},
"permissionsInfo": {
"view": [
{
"@type": "lowLevel",
"name": "StreamingSource",
"permissions": [
"read"
]
}
],
"manage": [
{
"@type": "lowLevel",
"name": "StreamingSource",
"permissions": [
"write"
]
}
]
}
}'
Response
A successful response returns the details of the queried flow specification, including its updated list of sourceConnectionSpecIds.
{
"id": "e77fde5a-22a8-11ed-861d-0242ac120002",
"updatedAt": 1633393222979,
"updatedBy": "1633393222979",
"updatedClient": "{UPDATED_CLIENT}",
"sandboxId": "{SANDBOX_ID}",
"sandboxName": "{SANDBOX_NAME}",
"imsOrgId": "{ORG_ID}",
"name": "GenericStreamingAEP",
"providerId": "521eee4d-8cbe-4906-bb48-fb6bd4450033",
"version": "1.0",
"sourceConnectionSpecIds": [
"e77fd9d2-22a8-11ed-861d-0242ac120002"
],
"targetConnectionSpecIds": [
"c604ff05-7f1a-43c0-8e18-33bf874cb11c"
],
"optionSpec": {
"name": "OptionSpec",
"spec": {
"$schema": "http://json-schema.org/draft-07/schema#",
"type": "object",
"properties": {
"errorDiagnosticsEnabled": {
"title": "Error diagnostics.",
"description": "Flag to enable detailed and sample error diagnostics summary.",
"type": "boolean",
"default": false
},
"partialIngestionPercent": {
"title": "Partial ingestion threshold.",
"description": "Percentage which defines the threshold of errors allowed before the run is marked as failed.",
"type": "number",
"exclusiveMinimum": 0
},
"inletUrl": {
"title": "Inlet Url.",
"description": "Inlet Url which defines the streaming endpoint.",
"type": "string"
}
}
}
},
"transformationSpecs": [
{
"name": "Mapping",
"spec": {
"$schema": "http://json-schema.org/draft-07/schema#",
"type": "object",
"description": "defines various params required for different mapping from source to target",
"properties": {
"mappingId": {
"type": "string"
},
"mappingVersion": {
"type": "string"
}
}
}
}
],
"attributes": {
"isSourceFlow": true,
"flacValidationSupported": true,
"frequency": "streaming",
"uiAttributes": {
"apiFeatures": {
"updateSupported": true,
"flowRunsSupported": true
}
}
},
"permissionsInfo": {
"manage": [
{
"@type": "lowLevel",
"name": "StreamingSource",
"permissions": [
"write"
]
}
],
"view": [
{
"@type": "lowLevel",
"name": "StreamingSource",
"permissions": [
"read"
]
}
]
}
}
]
}
Next steps
With your new connection specification added to the appropriate flow specification, you can now proceed to testing and submitting your new source. See the guide on testing and submitting a new source for more information.