Create an Amazon S3 base connection using the Flow Service API

A base connection represents the authenticated connection between a source and Adobe Experience Platform.

This tutorial walks you through the steps to create a base connection for Amazon S3 (hereinafter referred to as “S3”) using the Flow Service API.

Getting started

This guide requires a working understanding of the following components of Adobe Experience Platform:

  • Sources: Experience Platform allows data to be ingested from various sources while providing you with the ability to structure, label, and enhance incoming data using Platform services.
  • Sandboxes: Experience Platform provides virtual sandboxes which partition a single Platform instance into separate virtual environments to help develop and evolve digital experience applications.

The following sections provide additional information that you will need to know in order to successfully connect to an S3 storage using the Flow Service API.

Gather required credentials

In order for Flow Service to connect with your S3 storage, you must provide values for the following connection properties:

Credential Description
s3AccessKey The access key ID for your S3 bucket.
s3SecretKey The secret key ID for your S3 bucket.
serviceUrl (Optional) The custom S3 endpoint to connect to. This field is required when your S3 bucket is region-specific. The format for serviceUrl is: https://s3.{REGION}.amazonaws.com/).
bucketName The S3 bucket contains your data and its corresponding descriptive metadata. Your S3 bucket name must be between three and 63 characters long and must begin and end with either a letter or a number. The bucket name can only have lowercase letters, numbers, or hyphens (-), and cannot be formatted as an IP address.
folderPath The path to the folder in your S3 bucket where your data is stored. This credential is required when the user has restricted access.
s3SessionToken (Optional) A short-term, temporary token that allows you to provide temporary access to your S3 resources to users in untrusted environments. See the S3 overview for more information.
connectionSpec.id The connection specification returns a source’s connector properties, including authentication specifications related to creating the base and source connections. The connection specification ID for S3 is: ecadc60c-7455-4d87-84dc-2a0e293d997b.

For more information on getting started, visit this Amazon Web Services document.

Using Platform APIs

For information on how to successfully make calls to Platform APIs, see the guide on getting started with Platform APIs.

Create a base connection

A base connection retains information between your source and Platform, including your source’s authentication credentials, the current state of the connection, and your unique base connection ID. The base connection ID allows you to explore and navigate files from within your source and identify the specific items that you want to ingest, including information regarding their data types and formats.

To create a base connection ID, make a POST request to the /connections endpoint while providing your S3 authentication credentials as part of the request parameters.

API format

POST /connections

Request

The following request creates a base connection for S3:

curl -X POST \
    'https://platform.adobe.io/data/foundation/flowservice/connections' \
    -H 'Authorization: Bearer {ACCESS_TOKEN}' \
    -H 'x-api-key: {API_KEY}' \
    -H 'x-gw-ims-org-id: {ORG_ID}' \
    -H 'x-sandbox-name: {SANDBOX_NAME}' \
    -H 'Content-Type: application/json' \
    -d '{
        "name": "Amazon S3 base connection",
        "description": "Amazon S3 base connection with temporary session token",
        "auth": {
            "specName": "Access Key",
            "params": {
                "s3AccessKey": "{S3_ACCESS_KEY}",
                "s3SecretKey": "{S3_SECRET_KEY}",
                "s3SessionToken": "{S3_SESSION_TOKEN}
            }
        },
        "connectionSpec": {
            "id": "ecadc60c-7455-4d87-84dc-2a0e293d997b",
            "version": "1.0"
        }
    }'
Property Description
auth.params.s3AccessKey The access key associated with your S3 bucket.
auth.params.s3SecretKey Your secret key associated with your S3 bucket.
auth.params.s3SessionToken (Optional) The short-term, temporary S3 token used to access your bucket.
connectionSpec.id The S3 connection specification ID: ecadc60c-7455-4d87-84dc-2a0e293d997b

Response

A successful response returns details of the newly created connection, including its unique identifier (id). This ID is required to explore your storage in the next tutorial.

{
    "id": "4cb0c374-d3bb-4557-b139-5712880adc55",
    "etag": "\"1700d77b-0000-0200-0000-5e3b41a10000\""
}

Next steps

By following this tutorial, you have created an S3 connection using APIs and a unique ID was obtained as part of the response body. You can use this connection ID to explore cloud storages using the Flow Service API.

Appendix

The following section provides additional information on using the S3 source.

Update your S3 session token

The s3SessionToken is temporary and must be updated when it expires. You can update the session token associated with your base connection by making a PATCH request to Flow Service API. See the S3 overview for more information on temporary security credentials for S3.

IMPORTANT

The If-Match header is required when making a PATCH request. The value for this header is the unique etag of the connection you want to update.

API format

PATCH /connections

Request

The following request creates a base connection for S3:

curl -X PATCH \
    'https://platform.adobe.io/data/foundation/flowservice/connections/4cb0c374-d3bb-4557-b139-5712880adc55' \
    -H 'Authorization: Bearer {ACCESS_TOKEN}' \
    -H 'Content-Type: application/json' \
    -H 'x-api-key: {API_KEY}' \
    -H 'x-gw-ims-org-id: {ORG_ID}' \
    -H 'x-sandbox-name: {SANDBOX_NAME}' \
    -H 'If-Match: "1700d77b-0000-0200-0000-5e3b41a10000"'
    -d '[
        {
            "op": "replace",
            "path": "/auth/params/s3SessionToken",
            "value": "{SESSION_TOKEN}"
        }
    ]'
Parameter Description
op The operation call used to define the action needed to update the connection. Operations include: add, replace, and remove.
path The path of the parameter to be updated.
value The new value you want to update your parameter with.

Response

A successful response returns your base connection ID and an updated etag. You can verify the update by making a GET request to the Flow Service API, while providing your connection ID.

{
    "id": "4cb0c374-d3bb-4557-b139-5712880adc55",
    "etag": "\"3600e378-0000-0200-0000-5f40212f0000\""
}

Business.Adobe.com resources

Docs links
Campaign Management Device Graph Tag Management Data Management Platform Marketing Analytics Search Marketing Management

On this page