Create a new connection specification using the Flow Service API
A connection specification represents the structure of a source. It contains information on a source’s authentication requirements, defines how source data can be explored and inspected, and provides information on the attributes of a given source. The /connectionSpecs endpoint in the Flow Service API allows you to programmatically manage the connection specifications within your organization.
The following document provides steps on how to create a connection specification using the Flow Service API and integrate a new source through Self-Serve Sources (Batch SDK).
Getting started
Before continuing, please review the getting started guide for links to related documentation, a guide to reading the sample API calls in this document, and important information regarding required headers that are needed to successfully make calls to any Experience Platform API.
Collect artifacts
In order to create a new batch source using Self-Serve Sources, you must first coordinate with Adobe, request a private Git repository, and align with Adobe on the details regarding the label, description, category, and icon for your source.
Once provided, you must structure your private Git repository like so:
-
Sources
-
-
Artifacts
- {your_source}-category.txt
- {your_source}-description.txt
- {your_source}-icon.svg
- {your_source}-label.txt
- {your_source}-connectionSpec.json
-
-
| Artifacts (file names) | Description | Example |
|---|---|---|
| The name of your source. This folder should contain all artifacts related to your source, within your private Git repository. | mailchimp-members | |
| {your_source}-category.txt |
The category to which your source belongs, formatted as a text file. The list of available source categories supported by Self-Serve Sources (Batch SDK) include:
Note: If you believe that your source does not fit in any of the above categories, please contact your Adobe representative to discuss. | mailchimp-members-category.txt Inside the file, please specify the category of your source, like: marketingAutomation. |
| {your_source}-description.txt | A brief description of your source. | Mailchimp Members is marketing automation source that you can use to bring Mailchimp Members data to Experience Platform. |
| {your_source}-icon.svg | The image to be used to represent your source in the Experience Platform sources catalog. This icon must be an SVG file. | |
| {your_source}-label.txt | The name of your source as it should appear in the Experience Platform sources catalog. | Mailchimp Members |
| {your_source}-connectionSpec.json | A JSON file that contains the connection specification of your source. This file is not initially required as you will be populating your connection specification as you complete this guide. | mailchimp-members-connectionSpec.json |
TIPDuring the testing period of your connection specification, in place of key values, you can use
text in the connection specification.Once you have added the necessary files to your private Git repository, you must then create a pull request (PR) for Adobe to review. When your PR is approved and merged, you will be provided with an ID that can be used for your connection specification to refer to your source’s label, description, and icon.
Next, follow the steps outlined below to configure your connection specification. For additional guidance on the different functionalities that you can add to your source, such as advanced scheduling, custom schema, or different pagination types, please review the guide on configuring source specifications.
Copy connection specification template
Once you have gathered the required artifacts, copy and paste the connection specification template below to the text editor of your choice and then update the attributes in brackets {} with information relevant to your specific source.
{
"name": "generic-rest-extension",
"type": "generic-rest",
"description": "{DESCRIPTION}",
"providerId": "0ed90a81-07f4-4586-8190-b40eccef1c5a",
"version": "1.0",
"attributes": {
"uiAttributes": {
"apiFeatures": {
"explorePaginationSupported": false
}
}
},
"authSpec": [
{
"name": "OAuth2 Refresh Code",
"type": "OAuth2RefreshCode",
"spec": {
"$schema": "http://json-schema.org/draft-07/schema#",
"type": "object",
"description": "Define auth params required for connecting to generic rest using oauth2 authorization code.",
"properties": {
"authorizationTestUrl": {
"description": "Authorization test url to validate accessToken.",
"type": "string"
},
"clientId": {
"description": "Client id of user account.",
"type": "string"
},
"clientSecret": {
"description": "Client secret of user account.",
"type": "string",
"format": "password"
},
"accessToken": {
"description": "Access Token",
"type": "string",
"format": "password"
},
"refreshToken": {
"description": "Refresh Token",
"type": "string",
"format": "password"
},
"expirationDate": {
"description": "Date of token expiry.",
"type": "string",
"format": "date",
"uiAttributes": {
"hidden": true
}
},
"accessTokenUrl": {
"description": "Access token url to fetch access token.",
"type": "string"
},
"requestParameterOverride": {
"type": "object",
"description": "Specify parameter to override.",
"properties": {
"accessTokenField": {
"description": "Access token field name to override.",
"type": "string"
},
"refreshTokenField": {
"description": "Refresh token field name to override.",
"type": "string"
},
"expireInField": {
"description": "ExpireIn field name to override.",
"type": "string"
},
"authenticationMethod": {
"description": "Authentication method override.",
"type": "string",
"enum": [
"GET",
"POST"
]
},
"clientId": {
"description": "ClientId field name override.",
"type": "string"
},
"clientSecret": {
"description": "ClientSecret field name override.",
"type": "string"
}
}
}
},
"required": [
"host",
"accessToken"
]
}
},
{
"name": "Basic Authentication",
"type": "BasicAuthentication",
"spec": {
"$schema": "http://json-schema.org/draft-07/schema#",
"type": "object",
"description": "defines auth params required for connecting to rest service.",
"properties": {
"host": {
"type": "string",
"description": "Enter resource url host path"
},
"username": {
"description": "Username to connect rest endpoint.",
"type": "string"
},
"password": {
"description": "Password to connect rest endpoint.",
"type": "string",
"format": "password"
}
},
"required": [
"host",
"username",
"password"
]
}
}
],
"sourceSpec": {
"attributes": {
"uiAttributes": {
"isSource": true,
"isPreview": true,
"isBeta": true,
"category": {
"key": "protocols"
},
"icon": {
"key": "genericRestIcon"
},
"description": {
"key": "genericRestDescription"
},
"label": {
"key": "genericRestLabel"
}
}
},
"spec": {
"$schema": "http://json-schema.org/draft-07/schema#",
"type": "object",
"description": "defines static and user input parameters to fetch resource values.",
"properties": {
"urlParams": {
"type": "object",
"properties": {
"host": {
"type": "string",
"description": "Enter resource url host path."
},
"path": {
"type": "string",
"description": "Enter resource path",
"example": "/3.0/reports/campaignId/email-activity"
},
"method": {
"type": "string",
"description": "Http method type.",
"enum": [
"GET",
"POST"
]
},
"queryParams": {
"type": "object",
"description": "query parameters in json format",
"example": "{'key':'value','key1':'value1'} in JSON format"
}
},
"required": [
"path",
"method"
]
},
"headerParams": {
"type": "object",
"description": "header parameters in json format",
"example": "{'user':'c26f50c88dc035610e6753f807e28e9','x-api-key':'apiKey'}"
},
"contentPath": {
"type": "object",
"description": "Params required for main collection content.",
"properties": {
"path": {
"description": "path to main content.",
"type": "string",
"example": "$.emails"
},
"skipAttributes": {
"type": "array",
"description": "list of attributes that needs to be skipped while fattening the array.",
"example": "[total_items]",
"items": {
"type": "string"
}
},
"keepAttributes": {
"type": "array",
"description": "list of attributes that needs to be kept while fattening the array.",
"example": "[total_items]",
"items": {
"type": "string"
}
},
"overrideWrapperAttribute": {
"type": "string",
"description": "rename root content path node.",
"example": "email"
}
},
"required": [
"path"
]
},
"explodeEntityPath": {
"type": "object",
"description": "Params required for sub-array content.",
"properties": {
"path": {
"description": "path to sub-array content.",
"type": "string",
"example": "$.email.activity"
},
"skipAttributes": {
"type": "array",
"description": "list of attributes that needs to be skipped while fattening sub-array.",
"example": "[total_items]",
"items": {
"type": "string"
}
},
"keepAttributes": {
"type": "array",
"description": "list of attributes that needs to be kept while fattening the sub-array.",
"example": "[total_items]",
"items": {
"type": "string"
}
},
"overrideWrapperAttribute": {
"type": "string",
"description": "rename root content path node.",
"example": "activity"
}
},
"required": [
"path"
]
},
"paginationParams": {
"type": "object",
"description": "Params required to fetch data using pagination.",
"properties": {
"type": {
"description": "Pagination fetch type.",
"type": "string",
"enum": [
"OFFSET",
"POINTER"
]
},
"limitName": {
"type": "string",
"description": "limit property name",
"example": "limit or count"
},
"limitValue": {
"type": "integer",
"description": "number of records per page to fetch.",
"example": "limit=10 or count=10"
},
"offsetName": {
"type": "string",
"description": "offset property name",
"example": "offset"
},
"pointerPath": {
"type": "string",
"description": "pointer property name",
"example": "pointer"
}
},
"required": [
"type",
"limitName",
"limitValue"
]
},
"scheduleParams": {
"type": "object",
"description": "Params required to fetch data for batch schedule.",
"properties": {
"scheduleStartParamName": {
"type": "string",
"description": "order property name to get the order by date."
},
"scheduleEndParamName": {
"type": "string",
"description": "order property name to get the order by date."
},
"scheduleStartParamFormat": {
"type": "string",
"description": "order property name to get the order by date.",
"example": "yyyy-MM-ddTHH:mm:ssZ"
},
"scheduleEndParamFormat": {
"type": "string",
"description": "order property name to get the order by date.",
"example": "yyyy-MM-ddTHH:mm:ssZ"
}
},
"required": [
"scheduleStartParamName",
"scheduleEndParamName"
]
}
},
"required": [
"urlParams",
"contentPath",
"paginationParams",
"scheduleParams"
]
}
},
"exploreSpec": {
"name": "Resource",
"type": "Resource",
"requestSpec": {
"$schema": "http://json-schema.org/draft-07/schema#",
"type": "object"
},
"responseSpec": {
"$schema": "http: //json-schema.org/draft-07/schema#",
"type": "object",
"properties": {
"format": {
"type": "string"
},
"schema": {
"type": "object",
"properties": {
"columns": {
"type": "array",
"items": {
"type": "object",
"properties": {
"name": {
"type": "string"
},
"type": {
"type": "string"
}
}
}
}
}
},
"data": {
"type": "array",
"items": {
"type": "object"
}
}
}
}
}
}
Create a connection specification
Once you have acquired the connection specification template, you can now start authoring a new connection specification by filling in the appropriate values that corresponds to your source.
A connection specification can be divided into three distinct parts: the authentication specifications, the source specifications, and the explore specifications.
See the following documents for instructions on how to populate the values of each part of a connection specification:
With your specification information updated, you can submit the new connection specification by making a POST request to the /connectionSpecs endpoint of the Flow Service API.
API format
POST /connectionSpecs
Request
The following request is an example of a fully-authored connection specification for a MailChimp source:
curl -X POST \
'https://platform.adobe.io/data/foundation/flowservice/connectionSpecs' \
-H 'Authorization: Bearer {ACCESS_TOKEN}' \
-H 'x-api-key: {API_KEY}' \
-H 'x-gw-ims-org-id: {ORG_ID}' \
-H 'x-sandbox-name: {SANDBOX_NAME}' \
-H 'Content-Type: application/json' \
-d '{
"name": "MailChimp Members source",
"description": "MailChimp Members source using generic-rest and SDK",
"type": "generic-rest",
"providerId": "0ed90a81-07f4-4586-8190-b40eccef1c5a",
"version": "1.0",
"attributes": {
"uiAttributes": {
"apiFeatures": {
"explorePaginationSupported": false
}
}
},
"authSpec": [
{
"name": "OAuth2 Refresh Code",
"type": "OAuth2RefreshCode",
"spec": {
"$schema": "http://json-schema.org/draft-07/schema#",
"type": "object",
"description": "Define auth params required for connecting to generic rest using oauth2 authorization code.",
"properties": {
"domain": {
"type": "string",
"description": "Enter domain name for host url"
},
"authorizationTestUrl": {
"description": "Authorization test url to validate accessToken.",
"type": "string"
},
"accessToken": {
"description": "Access Token of mailChimp endpoint.",
"type": "string",
"format": "password"
}
},
"required": [
"domain",
"accessToken"
]
}
},
{
"name": "Basic Authentication",
"type": "BasicAuthentication",
"spec": {
"$schema": "http://json-schema.org/draft-07/schema#",
"type": "object",
"description": "defines auth params required for connecting to rest service.",
"properties": {
"domain": {
"type": "string",
"description": "Enter domain name for host url"
},
"username": {
"description": "Username to connect mailChimp endpoint.",
"type": "string"
},
"password": {
"description": "Password to connect mailChimp endpoint.",
"type": "string",
"format": "password"
}
},
"required": [
"domain",
"username",
"password"
]
}
}
],
"sourceSpec": {
"attributes": {
"uiAttributes": {
"isSource": true,
"isPreview": true,
"isBeta": true,
"icon": {
"key": "mailchimpMembersIcon"
},
"description": {
"key": "mailchimpMembersDescription"
},
"label": {
"key": "mailchimpMembersLabel"
}
},
"urlParams": {
"host": "https://${domain}.api.mailchimp.com",
"path": "/3.0/lists/${listId}/members",
"method": "GET"
},
"contentPath": {
"path": "$.members",
"skipAttributes": [
"_links",
"total_items",
"list_id"
],
"overrideWrapperAttribute": "member"
},
"paginationParams": {
"type": "OFFSET",
"limitName": "count",
"limitValue": "100",
"offSetName": "offset",
"endConditionName": "$.hasMore",
"endConditionValue": "Const:false"
},
"scheduleParams": {
"scheduleStartParamName": "since_last_changed",
"scheduleEndParamName": "before_last_changed",
"scheduleStartParamFormat": "yyyy-MM-ddTHH:mm:ss:fffffffK",
"scheduleEndParamFormat": "yyyy-MM-ddTHH:mm:ss:fffffffK"
}
},
"spec": {
"$schema": "http://json-schema.org/draft-07/schema#",
"type": "object",
"description": "Define user input parameters to fetch resource values.",
"properties": {
"listId": {
"type": "string",
"description": "listId for which members need to fetch."
}
}
}
},
"exploreSpec": {
"name": "Resource",
"type": "Resource",
"requestSpec": {
"$schema": "http://json-schema.org/draft-07/schema#",
"type": "object"
},
"responseSpec": {
"$schema": "http: //json-schema.org/draft-07/schema#",
"type": "object",
"properties": {
"format": {
"type": "string"
},
"schema": {
"type": "object",
"properties": {
"columns": {
"type": "array",
"items": {
"type": "object",
"properties": {
"name": {
"type": "string"
},
"type": {
"type": "string"
}
}
}
}
}
},
"data": {
"type": "array",
"items": {
"type": "object"
}
}
}
}
}
}'
Response
A successful response returns the newly created connection specification, including its unique id.
{
"id": "f6c0de0c-0a42-4cd9-9139-8768bf2f1b55",
"createdAt": 1633388930134,
"updatedAt": 1633388930134,
"createdBy": "{CREATED_BY}",
"updatedBy": "{UPDATED_BY}",
"createdClient": "{CREATED_CLIENT}",
"updatedClient": "{UPDATED_CLIENT}",
"sandboxId": "{SANDBOX_ID}",
"sandboxName": "{SANDBOX_NAME}",
"imsOrgId": "{ORG_ID}",
"name": "MailChimp Members source",
"providerId": "0ed90a81-07f4-4586-8190-b40eccef1c5a",
"version": "1.0",
"type": "generic-rest",
"authSpec": [
{
"name": "OAuth2 Refresh Code",
"type": "OAuth2RefreshCode",
"spec": {
"$schema": "http://json-schema.org/draft-07/schema#",
"type": "object",
"description": "Define auth params required for connecting to generic rest using oauth2 authorization code.",
"properties": {
"host": {
"type": "string",
"description": "Enter resource url host path"
},
"authorizationTestUrl": {
"description": "Authorization test url to validate accessToken.",
"type": "string"
},
"accessToken": {
"description": "Access Token of mailChimp endpoint.",
"type": "string",
"format": "password"
}
},
"required": [
"host",
"accessToken"
]
}
},
{
"name": "Basic Authentication",
"type": "BasicAuthentication",
"spec": {
"$schema": "http://json-schema.org/draft-07/schema#",
"type": "object",
"description": "defines auth params required for connecting to rest service.",
"properties": {
"host": {
"type": "string",
"description": "Enter resource url host path."
},
"username": {
"description": "Username to connect mailChimp endpoint.",
"type": "string"
},
"password": {
"description": "Password to connect mailChimp endpoint.",
"type": "string",
"format": "password"
}
},
"required": [
"host",
"username",
"password"
]
}
}
],
"sourceSpec": {
"spec": {
"$schema": "http://json-schema.org/draft-07/schema#",
"type": "object",
"description": "Define user input parameters to fetch resource values.",
"properties": {
"listId": {
"type": "string",
"description": "listId for which members need to fetch."
}
}
},
"attributes": {
"uiAttributes": {
"isSource": true,
"isPreview": true,
"isBeta": true,
"icon": {
"key": "mailchimpMembersIcon"
},
"description": {
"key": "mailchimpMembersDescription"
},
"label": {
"key": "mailchimpMembersLabel"
}
},
"urlParams": {
"path": "/3.0/lists/${listId}/members",
"method": "GET"
},
"contentPath": {
"path": "$.members",
"skipAttributes": [
"_links",
"total_items",
"list_id"
],
"overrideWrapperAttribute": "member"
},
"paginationParams": {
"type": "OFFSET",
"limitName": "count",
"limitValue": "100",
"offSetName": "offset",
"endConditionName": "$.hasMore",
"endConditionValue": "Const:false"
},
"scheduleParams": {
"scheduleStartParamName": "since_last_changed",
"scheduleEndParamName": "before_last_changed",
"scheduleStartParamFormat": "yyyy-MM-ddTHH:mm:ss:fffffffK",
"scheduleEndParamFormat": "yyyy-MM-ddTHH:mm:ss:fffffffK"
}
}
},
"exploreSpec": {
"name": "Resource",
"type": "Resource",
"requestSpec": {
"$schema": "http://json-schema.org/draft-07/schema#",
"type": "object"
},
"responseSpec": {
"$schema": "http: //json-schema.org/draft-07/schema#",
"type": "object",
"properties": {
"format": {
"type": "string"
},
"schema": {
"type": "object",
"properties": {
"columns": {
"type": "array",
"items": {
"type": "object",
"properties": {
"name": {
"type": "string"
},
"type": {
"type": "string"
}
}
}
}
}
},
"data": {
"type": "array",
"items": {
"type": "object"
}
}
}
}
},
"attributes": {
"uiAttributes": {
"apiFeatures": {
"explorePaginationSupported": false
}
}
}
}