The Amazon Kinesis source is available in the sources catalog to users who have purchased Real-Time Customer Data Platform Ultimate.
Adobe Experience Platform provides native connectivity for cloud providers like AWS, Google Cloud Platform, and Azure. You can bring your data from these systems into Platform.
Cloud storage sources can bring your own data into Platform without the need to download, format, or upload. Ingested data can be formatted as XDM JSON, XDM Parquet, or delimited. Every step of the process is integrated into the Sources workflow. Platform allows you to bring in data from Amazon Kinesis in real time.
The scale factor for Kinesis must be increased if you need to ingest high volume data. Currently, maximum volume of data that you can bring from your Kinesis account to Platform is 4000 records per second. To scale up and ingest higher volume data, please contact your Adobe representative.
The following section provides further information on prerequisite set up required before you can create a Kinesis source connection.
A Kinesis stream requires the following permissions to create a source connection:
These permissions are arranged through the Kinesis console and are checked by Platform once you enter your credentials and select your data stream.
The example below displays the minimum access rights required to create a Kinesis source connection.
|An action required to traverse through records.
|An action required to get records from a specific offset or shard ID.
|An action that returns information regarding the stream including the shard map, which is needed to generate a shard ID.
|An action required to list out available streams that you can select from the UI.
For more information on controlling access for Kinesis data streams, see the following Kinesis document.
Kinesis supports the following iterator types to allow you to specify the order of how your data is read:
|The data is read starting from a position identified by a specific sequence number.
|The data is read starting after position identified by a specific sequence number.
|The data is read starting from a position identified by a specific timestamp.
|The data is read starting from the oldest data record.
|The data is read starting from the most recent data record.
A Kinesis UI source currently only supports
TRIM_HORIZON, while the API supports both
LATEST as modes to get data. The default iterator value that Platform uses for the Kinesis source is
For more information on iterator types, see the following Kinesis document.
The documentation below provides information on how to connect Amazon Kinesis to Platform using APIs or the user interface: