DocumentationExperience PlatformSource Connectors Guide

[Ultimate]{class="badge positive"}

Amazon Kinesis source

Last update: Thu Jun 08 2023 00:00:00 GMT+0000 (Coordinated Universal Time)

CREATED FOR:

  • Developer
IMPORTANT
The Amazon Kinesis source is available in the sources catalog to users who have purchased Real-Time Customer Data Platform Ultimate.

Adobe Experience Platform provides native connectivity for cloud providers like AWS, Google Cloud Platform, and Azure. You can bring your data from these systems into Platform.

Cloud storage sources can bring your own data into Platform without the need to download, format, or upload. Ingested data can be formatted as XDM JSON, XDM Parquet, or delimited. Every step of the process is integrated into the Sources workflow. Platform allows you to bring in data from Amazon Kinesis in real time.

NOTE
The scale factor for Kinesis must be increased if you need to ingest high volume data. Currently, maximum volume of data that you can bring from your Kinesis account to Platform is 4000 records per second. To scale up and ingest higher volume data, please contact your Adobe representative.

Prerequisites

The following section provides further information on prerequisite set up required before you can create a Kinesis source connection.

Set up access policy

A Kinesis stream requires the following permissions to create a source connection:

  • GetShardIterator
  • GetRecords
  • DescribeStream
  • ListStreams

These permissions are arranged through the Kinesis console and are checked by Platform once you enter your credentials and select your data stream.

The example below displays the minimum access rights required to create a Kinesis source connection.

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "kinesis:GetShardIterator",
                "kinesis:GetRecords",
                "kinesis:DescribeStream",
                "kinesis:ListStreams"
            ],
            "Resource": [
                "arn:aws:kinesis:us-east-2:901341027596:stream/*"
            ]
        }
    ]
}
Property
Description
kinesis:GetShardIterator
An action required to traverse through records.
kinesis:GetRecords
An action required to get records from a specific offset or shard ID.
kinesis:DescribeStream
An action that returns information regarding the stream including the shard map, which is needed to generate a shard ID.
kinesis:ListStreams
An action required to list out available streams that you can select from the UI.

For more information on controlling access for Kinesis data streams, see the following Kinesis document.

Configure iterator type

Kinesis supports the following iterator types to allow you to specify the order of how your data is read:

Iterator type
Description
AT_SEQUENCE_NUMBER
The data is read starting from a position identified by a specific sequence number.
AFTER_SEQUENCE_NUMBER
The data is read starting after position identified by a specific sequence number.
AT_TIMESTAMP
The data is read starting from a position identified by a specific timestamp.
TRIM_HORIZON
The data is read starting from the oldest data record.
LATEST
The data is read starting from the most recent data record.

A Kinesis UI source currently only supports TRIM_HORIZON, while the API supports both TRIM_HORIZON and LATEST as modes to get data. The default iterator value that Platform uses for the Kinesis source is TRIM_HORIZON.

For more information on iterator types, see the following Kinesis document.

Connect Amazon Kinesis to Platform

The documentation below provides information on how to connect Amazon Kinesis to Platform using APIs or the user interface:

Using APIs

Using the UI

recommendation-more-help
337b99bb-92fb-42ae-b6b7-c7042161d089