Create a new connection specification using the Flow Service API
A connection specification represents the structure of a source. It contains information on a source’s authentication requirements, defines how source data can be explored and inspected, and provides information on the attributes of a given source. The /connectionSpecs endpoint in the Flow Service API allows you to programmatically manage the connection specifications within your organization.
The following document provides steps on how to create a connection specification using the Flow Service API and integrate a new source through Self-Serve Sources (Batch SDK).
Getting started
Before continuing, please review the getting started guide for links to related documentation, a guide to reading the sample API calls in this document, and important information regarding required headers that are needed to successfully make calls to any Experience Platform API.
Collect artifacts
In order to create a new batch source using Self-Serve Sources, you must first coordinate with Adobe, request a private Git repository, and align with Adobe on the details regarding the label, description, category, and icon for your source.
Once provided, you must structure your private Git repository like so:
-
Sources
-
-
Artifacts
- {your_source}-category.txt
- {your_source}-description.txt
- {your_source}-icon.svg
- {your_source}-label.txt
- {your_source}-connectionSpec.json
-
-
| Artifacts (file names) | Description | Example |
|---|---|---|
| The name of your source. This folder should contain all artifacts related to your source, within your private Git repository. | mailchimp-members | |
| {your_source}-category.txt |
The category to which your source belongs, formatted as a text file. The list of available source categories supported by Self-Serve Sources (Batch SDK) include:
Note: If you believe that your source does not fit in any of the above categories, please contact your Adobe representative to discuss. | mailchimp-members-category.txt Inside the file, please specify the category of your source, like: marketingAutomation. |
| {your_source}-description.txt | A brief description of your source. | Mailchimp Members is marketing automation source that you can use to bring Mailchimp Members data to Experience Platform. |
| {your_source}-icon.svg | The image to be used to represent your source in the Experience Platform sources catalog. This icon must be an SVG file. | |
| {your_source}-label.txt | The name of your source as it should appear in the Experience Platform sources catalog. | Mailchimp Members |
| {your_source}-connectionSpec.json | A JSON file that contains the connection specification of your source. This file is not initially required as you will be populating your connection specification as you complete this guide. | mailchimp-members-connectionSpec.json |
TIPtext in the connection specification.Once you have added the necessary files to your private Git repository, you must then create a pull request (PR) for Adobe to review. When your PR is approved and merged, you will be provided with an ID that can be used for your connection specification to refer to your source’s label, description, and icon.
Next, follow the steps outlined below to configure your connection specification. For additional guidance on the different functionalities that you can add to your source, such as advanced scheduling, custom schema, or different pagination types, please review the guide on configuring source specifications.
Copy connection specification template
Once you have gathered the required artifacts, copy and paste the connection specification template below to the text editor of your choice and then update the attributes in brackets {} with information relevant to your specific source.
{
"name": "generic-rest-extension",
"type": "generic-rest",
"description": "{DESCRIPTION}",
"providerId": "0ed90a81-07f4-4586-8190-b40eccef1c5a",
"version": "1.0",
"attributes": {
"uiAttributes": {
"apiFeatures": {
"explorePaginationSupported": false
}
}
},
"authSpec": [
{
"name": "OAuth2 Refresh Code",
"type": "OAuth2RefreshCode",
"spec": {
"$schema": "http://json-schema.org/draft-07/schema#",
"type": "object",
"description": "Define auth params required for connecting to generic rest using oauth2 authorization code.",
"properties": {
"authorizationTestUrl": {
"description": "Authorization test url to validate accessToken.",
"type": "string"
},
"clientId": {
"description": "Client id of user account.",
"type": "string"
},
"clientSecret": {
"description": "Client secret of user account.",
"type": "string",
"format": "password"
},
"accessToken": {
"description": "Access Token",
"type": "string",
"format": "password"
},
"refreshToken": {
"description": "Refresh Token",
"type": "string",
"format": "password"
},
"expirationDate": {
"description": "Date of token expiry.",
"type": "string",
"format": "date",
"uiAttributes": {
"hidden": true
}
},
"accessTokenUrl": {
"description": "Access token url to fetch access token.",
"type": "string"
},
"requestParameterOverride": {
"type": "object",
"description": "Specify parameter to override.",
"properties": {
"accessTokenField": {
"description": "Access token field name to override.",
"type": "string"
},
"refreshTokenField": {
"description": "Refresh token field name to override.",
"type": "string"
},
"expireInField": {
"description": "ExpireIn field name to override.",
"type": "string"
},
"authenticationMethod": {
"description": "Authentication method override.",
"type": "string",
"enum": [
"GET",
"POST"
]
},
"clientId": {
"description": "ClientId field name override.",
"type": "string"
},
"clientSecret": {
"description": "ClientSecret field name override.",
"type": "string"
}
}
}
},
"required": [
"host",
"accessToken"
]
}
},
{
"name": "Basic Authentication",
"type": "BasicAuthentication",
"spec": {
"$schema": "http://json-schema.org/draft-07/schema#",
"type": "object",
"description": "defines auth params required for connecting to rest service.",
"properties": {
"host": {
"type": "string",
"description": "Enter resource url host path"
},
"username": {
"description": "Username to connect rest endpoint.",
"type": "string"
},
"password": {
"description": "Password to connect rest endpoint.",
"type": "string",
"format": "password"
}
},
"required": [
"host",
"username",
"password"
]
}
}
],
"sourceSpec": {
"attributes": {
"uiAttributes": {
"isSource": true,
"isPreview": true,
"isBeta": true,
"category": {
"key": "protocols"
},
"icon": {
"key": "genericRestIcon"
},
"description": {
"key": "genericRestDescription"
},
"label": {
"key": "genericRestLabel"
}
}
},
"spec": {
"$schema": "http://json-schema.org/draft-07/schema#",
"type": "object",
"description": "defines static and user input parameters to fetch resource values.",
"properties": {
"urlParams": {
"type": "object",
"properties": {
"host": {
"type": "string",
"description": "Enter resource url host path."
},
"path": {
"type": "string",
"description": "Enter resource path",
"example": "/3.0/reports/campaignId/email-activity"
},
"method": {
"type": "string",
"description": "Http method type.",
"enum": [
"GET",
"POST"
]
},
"queryParams": {
"type": "object",
"description": "query parameters in json format",
"example": "{'key':'value','key1':'value1'} in JSON format"
}
},
"required": [
"path",
"method"
]
},
"headerParams": {
"type": "object",
"description": "header parameters in json format",
"example": "{'user':'c26f50c88dc035610e6753f807e28e9','x-api-key':'apiKey'}"
},
"contentPath": {
"type": "object",
"description": "Params required for main collection content.",
"properties": {
"path": {
"description": "path to main content.",
"type": "string",
"example": "$.emails"
},
"skipAttributes": {
"type": "array",
"description": "list of attributes that needs to be skipped while fattening the array.",
"example": "[total_items]",
"items": {
"type": "string"
}
},
"keepAttributes": {
"type": "array",
"description": "list of attributes that needs to be kept while fattening the array.",
"example": "[total_items]",
"items": {
"type": "string"
}
},
"overrideWrapperAttribute": {
"type": "string",
"description": "rename root content path node.",
"example": "email"
}
},
"required": [
"path"
]
},
"explodeEntityPath": {
"type": "object",
"description": "Params required for sub-array content.",
"properties": {
"path": {
"description": "path to sub-array content.",
"type": "string",
"example": "$.email.activity"
},
"skipAttributes": {
"type": "array",
"description": "list of attributes that needs to be skipped while fattening sub-array.",
"example": "[total_items]",
"items": {
"type": "string"
}
},
"keepAttributes": {
"type": "array",
"description": "list of attributes that needs to be kept while fattening the sub-array.",
"example": "[total_items]",
"items": {
"type": "string"
}
},
"overrideWrapperAttribute": {
"type": "string",
"description": "rename root content path node.",
"example": "activity"
}
},
"required": [
"path"
]
},
"paginationParams": {
"type": "object",
"description": "Params required to fetch data using pagination.",
"properties": {
"type": {
"description": "Pagination fetch type.",
"type": "string",
"enum": [
"OFFSET",
"POINTER"
]
},
"limitName": {
"type": "string",
"description": "limit property name",
"example": "limit or count"
},
"limitValue": {
"type": "integer",
"description": "number of records per page to fetch.",
"example": "limit=10 or count=10"
},
"offsetName": {
"type": "string",
"description": "offset property name",
"example": "offset"
},
"pointerPath": {
"type": "string",
"description": "pointer property name",
"example": "pointer"
}
},
"required": [
"type",
"limitName",
"limitValue"
]
},
"scheduleParams": {
"type": "object",
"description": "Params required to fetch data for batch schedule.",
"properties": {
"scheduleStartParamName": {
"type": "string",
"description": "order property name to get the order by date."
},
"scheduleEndParamName": {
"type": "string",
"description": "order property name to get the order by date."
},
"scheduleStartParamFormat": {
"type": "string",
"description": "order property name to get the order by date.",
"example": "yyyy-MM-ddTHH:mm:ssZ"
},
"scheduleEndParamFormat": {
"type": "string",
"description": "order property name to get the order by date.",
"example": "yyyy-MM-ddTHH:mm:ssZ"
}
},
"required": [
"scheduleStartParamName",
"scheduleEndParamName"
]
}
},
"required": [
"urlParams",
"contentPath",
"paginationParams",
"scheduleParams"
]
}
},
"exploreSpec": {
"name": "Resource",
"type": "Resource",
"requestSpec": {
"$schema": "http://json-schema.org/draft-07/schema#",
"type": "object"
},
"responseSpec": {
"$schema": "http: //json-schema.org/draft-07/schema#",
"type": "object",
"properties": {
"format": {
"type": "string"
},
"schema": {
"type": "object",
"properties": {
"columns": {
"type": "array",
"items": {
"type": "object",
"properties": {
"name": {
"type": "string"
},
"type": {
"type": "string"
}
}
}
}
}
},
"data": {
"type": "array",
"items": {
"type": "object"
}
}
}
}
}
}