This tutorial walks you through the steps to connect Azure Event Hubs (hereinafter referred to as “Event Hubs”) to Experience Platform, using the Flow Service API.
This guide requires a working understanding of the following components of Adobe Experience Platform:
The following sections provide additional information that you will need to know in order to successfully connect Event Hubs to Platform using the Flow Service API.
In order for Flow Service to connect with your Event Hubs account, you must provide values for the following connection properties:
Credential | Description |
---|---|
sasKeyName |
The name of the authorization rule, which is also known as the SAS key name. |
sasKey |
The primary key of the Event Hubs namespace. The sasPolicy that the sasKey corresponds to must have manage rights configured in order for the Event Hubs list to be populated. |
namespace |
The namespace of the Event Hubs you are accessing. An Event Hubs namespace provides a unique scoping container, in which you can create one or more Event Hubs. |
connectionSpec.id |
The connection specification returns a source’s connector properties, including authentication specifications related to creating the base and source connections. The Event Hubs connection specification ID is: bf9f5905-92b7-48bf-bf20-455bc6b60a4e . |
For more information about these values, refer to this Event Hubs document.
For information on how to successfully make calls to Platform APIs, see the guide on getting started with Platform APIs.
The first step in creating a source connection is to authenticate your Event Hubs source and generate a base connection ID. A base connection ID allows you to explore and navigate files from within your source and identify specific items that you want to ingest, including information regarding their data types and formats.
To create a base connection ID, make a POST request to the /connections
endpoint while providing your Event Hubs authentication credentials as part of the request parameters.
API format
POST /connections
Request
curl -X POST \
'https://platform.adobe.io/data/foundation/flowservice/connections' \
-H 'Authorization: Bearer {ACCESS_TOKEN}' \
-H 'x-api-key: {API_KEY}' \
-H 'x-gw-ims-org-id: {ORG_ID}' \
-H 'x-sandbox-name: {SANDBOX_NAME}' \
-H 'Content-Type: application/json' \
-d '{
"name": "Azure Event Hubs connection",
"description": "Connector for Azure Event Hubs",
"auth": {
"specName": "Azure EventHub authentication credentials",
"params": {
"sasKeyName": "{SAS_KEY_NAME}",
"sasKey": "{SAS_KEY}",
"namespace": "{NAMESPACE}"
}
},
"connectionSpec": {
"id": "bf9f5905-92b7-48bf-bf20-455bc6b60a4e",
"version": "1.0"
}
}'
Property | Description |
---|---|
auth.params.sasKeyName |
The name of the authorization rule, which is also known as the SAS key name. |
auth.params.sasKey |
The generated shared access signature. |
auth.params.namespace |
The namespace of the Event Hubs you are accessing. |
connectionSpec.id |
The Event Hubs connection specification ID is: bf9f5905-92b7-48bf-bf20-455bc6b60a4e |
Response
A successful response returns details of the newly created base connection, including its unique identifier (id
). This connection ID is required in the next step to create a source connection.
{
"id": "4cdbb15c-fb1e-46ee-8049-0f55b53378fe",
"etag": "\"6507cfd8-0000-0200-0000-5e18fc600000\""
}
A source connection creates and manages the connection to the external source from where data is ingested. A source connection consists of information like data source, data format, and a source connection ID needed to create a dataflow. A source connection instance is specific to a tenant and IMS Organization.
To create a source connection, make a POST request to the /sourceConnections
endpoint of the Flow Service API.
API format
POST /sourceConnections
Request
curl -X POST \
'https://platform.adobe.io/data/foundation/flowservice/sourceConnections' \
-H 'authorization: Bearer {ACCESS_TOKEN}' \
-H 'content-type: application/json' \
-H 'x-api-key: {API_KEY}' \
-H 'x-gw-ims-org-id: {ORG_ID}' \
-H 'x-sandbox-name: {SANDBOX_NAME}' \
-d '{
"name": "Azure Event Hubs source connection",
"description": "A source connection for Azure Event Hubs",
"baseConnectionId": "4cdbb15c-fb1e-46ee-8049-0f55b53378fe",
"connectionSpec": {
"id": "bf9f5905-92b7-48bf-bf20-455bc6b60a4e",
"version": "1.0"
},
"data": {
"format": "json"
},
"params": {
"eventHubName": "{EVENTHUB_NAME}",
"dataType": "raw",
"reset": "latest",
"consumerGroup": "{CONSUMER_GROUP}"
}
}'
Property | Description |
---|---|
name |
The name of your source connection. Ensure that the name of your source connection is descriptive as you can use this to look up information on your source connection. |
description |
An optional value that you can provide to include more information on your source connection. |
baseConnectionId |
The connection ID of your Event Hubs source that was generated in the previous step. |
connectionSpec.id |
The fixed connection specification ID for Event Hubs. This ID is: bf9f5905-92b7-48bf-bf20-455bc6b60a4e . |
data.format |
The format of the Event Hubs data that you want to ingest. Currently, the only supported data format is json . |
params.eventHubName |
The name for your Event Hubs source. |
params.dataType |
This parameter defines the type of the data that is being ingested. Supported data types include: raw and xdm . |
params.reset |
This parameter defines how the data will be read. Use latest to start reading from the most recent data, and use earliest to start reading from the first available data in the stream. This parameter is optional and defaults to earliest if unprovided. |
params.consumerGroup |
The publish or subscription mechanism to be used for Event Hubs. This parameter is optional and defaults to $Default if unprovided. Refer to this Event Hubs guide on event consumers for more information. |
By following this tutorial, you have created an Event Hubs source connection using the Flow Service API. You can use this source connection ID in the next tutorial to create a streaming dataflow using the Flow Service API.