Connect Amazon S3 to Experience Platform using the Flow Service API
- Topics:
- Sources
CREATED FOR:
- Developer
Read this guide to learn how you can connect your Amazon S3 source account to Adobe Experience Platform using the Flow Service API.
Get started
This guide requires a working understanding of the following components of Adobe Experience Platform:
- Sources: Experience Platform allows data to be ingested from various sources while providing you with the ability to structure, label, and enhance incoming data using Experience Platform services.
- Sandboxes: Experience Platform provides virtual sandboxes which partition a single Experience Platform instance into separate virtual environments to help develop and evolve digital experience applications.
Using Experience Platform APIs
For information on how to successfully make calls to Experience Platform APIs, see the guide on getting started with Experience Platform APIs.
Connect Amazon S3 to Experience Platform on Azure
Read the steps below for information on how to connect your Amazon S3 source to Experience Platform on Azure.
Gather required credentials
In order for Flow Service to connect with your Amazon S3 storage, you must provide values for the following connection properties:
s3AccessKey
s3SecretKey
serviceUrl
serviceUrl
is: https://s3.{REGION}.amazonaws.com/)
.bucketName
-
), and cannot be formatted as an IP address.folderPath
s3SessionToken
connectionSpec.id
ecadc60c-7455-4d87-84dc-2a0e293d997b
.For more information on getting started, visit this Amazon Web Services document.
Create a base connection for Amazon S3 on Experience Platform on Azure
A base connection retains information between your source and Experience Platform, including your source’s authentication credentials, the current state of the connection, and your unique base connection ID. The base connection ID allows you to explore and navigate files from within your source and identify the specific items that you want to ingest, including information regarding their data types and formats.
To create a base connection ID, make a POST request to the /connections
endpoint while providing your S3 authentication credentials as part of the request parameters.
API format
POST /connections
Request
The following request creates a base connection for Amazon S3:
Select to view request example
curl -X POST \
'https://platform.adobe.io/data/foundation/flowservice/connections' \
-H 'Authorization: Bearer {ACCESS_TOKEN}' \
-H 'x-api-key: {API_KEY}' \
-H 'x-gw-ims-org-id: {ORG_ID}' \
-H 'x-sandbox-name: {SANDBOX_NAME}' \
-H 'Content-Type: application/json' \
-d '{
"name": "Amazon S3 base connection",
"description": "Amazon S3 base connection with temporary session token",
"auth": {
"specName": "Access Key",
"params": {
"s3AccessKey": "{S3_ACCESS_KEY}",
"s3SecretKey": "{S3_SECRET_KEY}",
"s3SessionToken": "{S3_SESSION_TOKEN}
}
},
"connectionSpec": {
"id": "ecadc60c-7455-4d87-84dc-2a0e293d997b",
"version": "1.0"
}
}'
auth.params.s3AccessKey
auth.params.s3SecretKey
auth.params.s3SessionToken
connectionSpec.id
ecadc60c-7455-4d87-84dc-2a0e293d997b
Response
A successful response returns details of the newly created connection, including its unique identifier (id
). This ID is required to explore your storage in the next tutorial.
{
"id": "4cb0c374-d3bb-4557-b139-5712880adc55",
"etag": "\"1700d77b-0000-0200-0000-5e3b41a10000\""
}
Update your S3 session token
The s3SessionToken
is temporary and must be updated when it expires. You can update the session token associated with your base connection by making a PATCH request to Flow Service API. See the S3 overview for more information on temporary security credentials for S3.
If-Match
header is required when making a PATCH request. The value for this header is the unique etag of the connection you want to update.API format
PATCH /connections
Request
The following request creates a base connection for Amazon S3:
curl -X PATCH \
'https://platform.adobe.io/data/foundation/flowservice/connections/4cb0c374-d3bb-4557-b139-5712880adc55' \
-H 'Authorization: Bearer {ACCESS_TOKEN}' \
-H 'Content-Type: application/json' \
-H 'x-api-key: {API_KEY}' \
-H 'x-gw-ims-org-id: {ORG_ID}' \
-H 'x-sandbox-name: {SANDBOX_NAME}' \
-H 'If-Match: "1700d77b-0000-0200-0000-5e3b41a10000"'
-d '[
{
"op": "replace",
"path": "/auth/params/s3SessionToken",
"value": "{SESSION_TOKEN}"
}
]'
op
add
, replace
, and remove
.path
value
Response
A successful response returns your base connection ID and an updated etag. You can verify the update by making a GET request to the Flow Service API, while providing your connection ID.
{
"id": "4cb0c374-d3bb-4557-b139-5712880adc55",
"etag": "\"3600e378-0000-0200-0000-5f40212f0000\""
}
Connect Amazon S3 to Experience Platform on Amazon Web Services (AWS)
Read the steps below for information on how to connect your Amazon S3 source to Experience Platform on AWS.
Prerequisites
To connect your Amazon S3 account to Experience Platform on AWS, you must have the following:
- An AWS account with access to the Amazon S3 bucket or folder that you want to connect.
- The necessary IAM permissions that allow
s3:GetObject
ands3:ListBucket
actions.
Retrieve the IAM role for your bucket permissions
API format
GET /connectionSpecs/{CONNECTION_SPEC_ID}/configs?authType={AUTH_TYPE}
Request
curl -X GET \
'https://platform.adobe.io/data/foundation/flowservice/connectionSpecs/ecadc60c-7455-4d87-84dc-2a0e293d997b/configs?authType=IamBasedAuthentication' \
-H 'Authorization: Bearer {ACCESS_TOKEN}' \
-H 'Content-Type: application/json' \
-H 'x-api-key: {API_KEY}' \
-H 'x-gw-ims-org-id: {ORG_ID}' \
-H 'x-sandbox-name: {SANDBOX_NAME}' \
Response
A successful response returns your IAM Role. This value is required in the next step to set up permissions in your Amazon S3 bucket.
{
"configParams": {
"IAMRole": "{IAM_ROLE}"
}
}
Set up permissions in your Amazon S3 bucket
- Log in to your account in the AWS Management Console.
- Navigate to your Amazon S3 bucket and then select Permissions.
- Edit the bucket policy and add the following permissions:
IAM_ROLE_TO_ALLOW_LIST
is the IAM role fetched through the API in the previous step. You must replace {YOUR_BUCKET_NAME} with the actual name of your Amazon S3 bucket. If you want to give access to a specific folder, then you must also replace {YOUR_FOLDER_NAME} with the actual name of the folder that you want to give access to.{
"Version": "2012-10-17",
"Statement": [
{
"Sid": "AEP Get Object Related Policy Prod",
"Effect": "Allow",
"Principal": {
"AWS": "{IAM_ROLE_TO_ALLOW_LIST}"
},
"Action": "s3:Get*",
"Resource": "arn:aws:s3:::{YOUR_BUCKET_NAME}/{YOUR_FOLDER_NAME}"
},
{
"Sid": "AEP List Bucket Prod",
"Effect": "Allow",
"Principal": {
"AWS": "{IAM_ROLE_TO_ALLOW_LIST}"
},
"Action": "s3:ListBucket",
"Resource": "arn:aws:s3:::{YOUR_BUCKET_NAME}"
}
]
}
Create a base connection for Amazon S3 on Experience Platform in AWS
API format
POST /connections
Request
The following request creates a base connection for Amazon S3:
curl -X POST \
'https://platform.adobe.io/data/foundation/flowservice/connections' \
-H 'Authorization: Bearer {ACCESS_TOKEN}' \
-H 'x-api-key: {API_KEY}' \
-H 'x-gw-ims-org-id: {ORG_ID}' \
-H 'x-sandbox-name: {SANDBOX_NAME}' \
-H 'Content-Type: application/json' \
-d '{
"name": "Amazon S3 base connection for Experience Platform on AWS",
"description": "Amazon S3 base connection for Experience Platform on AWS",
"auth": {
"specName": "IAMRole Based",
"params": {
"bucketName": "{YOUR_BUCKET_NAME}"
}
},
"connectionSpec": {
"id": "ecadc60c-7455-4d87-84dc-2a0e293d997b",
"version": "1.0"
}
}'
auth.params.bucketName
Response
A successful response returns details of the newly created connection, including its unique identifier (id
). This ID is required to explore your storage in the next tutorial.
{
"id": "4cb0c374-d3bb-4557-b139-5712880adc55",
"etag": "\"1700d77b-0000-0200-0000-5e3b41a10000\""
}
Next steps
By following this tutorial, you have created an S3 connection using APIs and a unique ID was obtained as part of the response body. You can use this connection ID to explore cloud storages using the Flow Service API.