Amazon Kinesis connector

Adobe Experience Platform provides native connectivity for cloud providers like AWS, Google Cloud Platform, and Azure. You can bring your data from these systems into Platform.

Cloud storage sources can bring your own data into Platform without the need to download, format, or upload. Ingested data can be formatted as XDM JSON, XDM Parquet, or delimited. Every step of the process is integrated into the Sources workflow. Platform allows you to bring in data from Amazon Kinesis in real time.

Prerequisites

The following section provides further information on prerequisite set up required before you can create a Kinesis source connection.

Set up access policy

A Kinesis stream requires the following permissions to create a source connection:

  • GetShardIterator
  • GetRecords
  • DescribeStream
  • ListStreams

These permissions are arranged through the Kinesis console and are checked by Platform once you enter your credentials and select your data stream.

The example below displays the minimum access rights required to create a Kinesis source connection.

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "kinesis:GetShardIterator",
                "kinesis:GetRecords",
                "kinesis:DescribeStream",
                "kinesis:ListStreams"
            ],
            "Resource": [
                "arn:aws:kinesis:us-east-2:901341027596:stream/*"
            ]
        }
    ]
}
Property Description
kinesis:GetShardIterator An action required to traverse through records.
kinesis:GetRecords An action required to get records from a specific offset or shard ID.
kinesis:DescribeStream An action that returns information regarding the stream including the shard map, which is needed to generate a shard ID.
kinesis:ListStreams An action required to list out available streams that you can select from the UI.

For more information on controlling access for Kinesis data streams, see the following Kinesis document.

Configure iterator type

Kinesis supports the following iterator types to allow you to specify the order of how your data is read:

Iterator type Description
AT_SEQUENCE_NUMBER The data is read starting from a position identified by a specific sequence number.
AFTER_SEQUENCE_NUMBER The data is read starting after position identified by a specific sequence number.
AT_TIMESTAMP The data is read starting from a position identified by a specific timestamp.
TRIM_HORIZON The data is read starting from the oldest data record.
LATEST The data is read starting from the most recent data record.

A Kinesis UI source currently only supports TRIM_HORIZON, while the API supports both TRIM_HORIZON and LATEST as modes to get data. The default iterator value that Platform uses for the Kinesis source is TRIM_HORIZON.

For more information on iterator types, see the following Kinesis document.

Connect Amazon Kinesis to Platform

The documentation below provides information on how to connect Amazon Kinesis to Platform using APIs or the user interface:

Using APIs

Using the UI

On this page