Amazon Kinesis connection

Overview

The Kinesis Data Streams service by Amazon Web Services allows you to collect and process large streams of data records in real time.

You can create a real-time outbound connection to your Amazon Kinesis storage to stream data from Adobe Experience Platform.

• For more information about Amazon Kinesis, see the Amazon documentation.
• To connect to Amazon Kinesis programmatically, see the Streaming destinations API tutorial.
• To connect to Amazon Kinesis using the Platform user interface, see the sections below.

Use cases

By using streaming destinations such as Amazon Kinesis, you can easily feed high-value segmentation events and associated profile attributes into your systems of choice.

For example, a prospect downloaded a white-paper which qualifies them into a “high-propensity to convert” segment. By mapping the segment that the prospect falls in to the Amazon Kinesis destination, you would receive this event in Amazon Kinesis. There, you can employ a do-it-yourself approach and describe business logic on top of the event, as you think would work best with your enterprise IT systems.

Export type and frequency

Refer to the table below for information about the destination export type and frequency.

IP address allowlist

To meet customers’ security and compliance requirements, Experience Platform provides a list of static IPs that you can allowlist for the Amazon Kinesis destination. Refer to IP address allow list for streaming destinations for the complete list of IPs to allowlist.

Required Amazon Kinesis permissions

To successfully connect and export data to your Amazon Kinesis streams, Experience Platform needs permissions for the following actions:

• kinesis:ListStreams
• kinesis:PutRecord
• kinesis:PutRecords

These permissions are arranged through the Kinesis console and are checked by Platform once you configure your Kinesis destination in the Platform user interface.

The example below displays the minimum access rights required to successfully export data to a Kinesis destination.

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "kinesis:ListStreams",
                "kinesis:PutRecord",
                "kinesis:PutRecords"
            ],
            "Resource": [
                "arn:aws:kinesis:us-east-2:901341027596:stream/*"
            ]
        }
    ]
}

For more information on controlling access for Kinesis data streams, read the following Kinesis document.

Connect to the destination

To connect to this destination, follow the steps described in the destination configuration tutorial. When connecting to this destination, you must provide the following information:

Authentication information

Input the fields below and select Connect to destination:

• Amazon Web Services access key and secret key: In Amazon Web Services, generate an access key - secret access key pair to grant Platform access to your Amazon Kinesis account. Learn more in the Amazon Web Services documentation.
• Region: Indicate which Amazon Web Services region to stream data to.

Fill in destination details

To configure details for the destination, fill in the required and optional fields below. An asterisk next to a field in the UI indicates that the field is required.

• Name: Provide a name for your connection to Amazon Kinesis
• Description: Provide a description for your connection to Amazon Kinesis.
• stream: Provide the name of an existing data stream in your Amazon Kinesis account. Platform will export data to this stream.
• Include Segment Names: Toggle if you want the data export to include the names of the segments you are exporting. For an example of a data export with this option selected, refer to the Exported data section further below.
• Include Segment Timestamps: Toggle if you want the data export to include the UNIX timestamp when the segments were created and updated, as well as the UNIX timestamp when the segments were mapped to the destination for activation. For an example of a data export with this option selected, refer to the Exported data section further below.

Enable alerts

You can enable alerts to receive notifications on the status of the dataflow to your destination. Select an alert from the list to subscribe to receive notifications on the status of your dataflow. For more information on alerts, see the guide on subscribing to destinations alerts using the UI.

When you are finished providing details for your destination connection, select Next.

Activate segments to this destination

See Activate audience data to streaming profile export destinations for instructions on activating audience segments to this destination.

Profile export behavior

Experience Platform optimizes the profile export behavior to your Amazon Kinesis destination, to only export data to your destination when relevant updates to a profile have occurred following segment qualification or other significant events. Profiles are exported to your destination in the following situations:

• The profile update was determined by a change in segment membership for at least one of the segments mapped to the destination. For example, the profile has qualified for one of the segments mapped to the destination or has exited one of the segments mapped to the destination.
• The profile update was determined by a change in the identity map. For example, a profile who had already qualified for one of the segments mapped to the destination has been added a new identity in the identity map attribute.
• The profile update was determined by a change in attributes for at least one of the attributes mapped to the destination. For example, one of the attributes mapped to the destination in the mapping step is added to a profile.

In all the cases described above, only the profiles where relevant updates have occurred are exported to your destination. For example, if a segment mapped to the destination flow has a hundred members, and five new profiles qualify for the segment, the export to your destination is incremental and only includes the five new profiles.

Note that the all the mapped attributes are exported for a profile, no matter where the changes lie. So, in the example above all the mapped attributes for those five new profiles will be exported even if the attributes themselves haven’t changed.

What determines a data export and what is included in the export

Regarding the data that is exported for a given profile, it is important to understand the two different concepts of what determines a data export to your Amazon Kinesis destination and which data is included in the export.

For example, consider this dataflow to an Amazon Kinesis destination where three segments are selected in the dataflow, and four attributes are mapped to the destination.

A profile export to the destination can be determined by a profile qualifying for or exiting one of the three mapped segments. However, in the data export, in the segmentMembership object (see Exported Data section below), other unmapped segments might appear, if that particular profile is a member of them. If a profile qualifies for the Customer with DeLorean Cars segment but is also a member of the Watched “Back to the Future” movie and Science fiction fans segments, then these other two segments will also be present in the segmentMembership object of the data export, even though these are not mapped in the dataflow.

From a profile attributes point of view, any changes to the four attributes mapped above will determine a destination export and any of the four mapped attributes present on the profile will be present in the data export.

Historical data backfill

When you add a new segment to an existing destination, or when you create a new destination and map segments to it, Experience Platform exports historical segment qualification data to the destination. Profiles which qualified for the segment before the segment was added to the destination are exported to the destination within approximately one hour.

Exported data

Your exported Experience Platform data lands in your Amazon Kinesis destination in JSON format. For example, the export below contains a profile that has qualified for a certain segment, is a member of another two segments, and exited another segment. The export also includes the profile attribute first name, last name, date of birth, and personal email address. The identities for this profile are ECID and email.

{
  "person": {
    "birthDate": "YYYY-MM-DD",
    "name": {
      "firstName": "John",
      "lastName": "Doe"
    }
  },
  "personalEmail": {
    "address": "john.doe@acme.com"
  },
  "segmentMembership": {
   "ups":{
      "7841ba61-23c1-4bb3-a495-00d3g5fe1e93":{
         "lastQualificationTime":"2022-01-11T21:24:39Z",
         "status":"exited"
      },
      "59bd2fkd-3c48-4b18-bf56-4f5c5e6967ae":{
         "lastQualificationTime":"2022-01-02T23:37:33Z",
         "status":"existing"
      },
      "947c1c46-008d-40b0-92ec-3af86eaf41c1":{
         "lastQualificationTime":"2021-08-25T23:37:33Z",
         "status":"existing"
      },
      "5114d758-ce71-43ba-b53e-e2a91d67b67f":{
         "lastQualificationTime":"2022-01-11T23:37:33Z",
         "status":"realized"
      }
   }
},
  "identityMap": {
    "ecid": [
      {
        "id": "14575006536349286404619648085736425115"
      },
      {
        "id": "66478888669296734530114754794777368480"
      }
    ],
    "email_lc_sha256": [
      {
        "id": "655332b5fa2aea4498bf7a290cff017cb4"
      },
      {
        "id": "66baf76ef9de8b42df8903f00e0e3dc0b7"
      }
    ]
  }
}

Below are further examples of exported data, depending on the UI settings you select in the connect destination flow for the Include Segment Names and Include Segment Timestamps options:

"segmentMembership": {
        "ups": {
          "5b998cb9-9488-4ec3-8d95-fa8338ced490": {
            "lastQualificationTime": "2019-04-15T02:41:50+0000",
            "status": "existing",
            "createdAt": 1648553325000,
            "updatedAt": 1648553330000,
            "mappingCreatedAt": 1649856570000,
            "mappingUpdatedAt": 1649856570000,
            "name": "First name equals John"
          }
        }
      }

"segmentMembership": {
        "ups": {
          "5b998cb9-9488-4ec3-8d95-fa8338ced490": {
            "lastQualificationTime": "2019-04-15T02:41:50+0000",
            "status": "existing",
            "createdAt": 1648553325000,
            "updatedAt": 1648553330000,
            "mappingCreatedAt": 1649856570000,
            "mappingUpdatedAt": 1649856570000,
          }
        }
      }

Limits and retry policy

In 95 percent of the time, Experience Platform attempts to offer a throughput latency of less than 10 minutes for successfully sent messages with a rate of less than 10.000 requests per second for each dataflow to an HTTP destination.

In case of failed requests to your HTTP API destination, Experience Platform stores the failed requests and retries twice to send the requests to your endpoint.

Business.Adobe.com resources