Data Streams

Our customer’s marketing organizations rely on timely and focused Marketing Campaigns to stay on top of their business and be competitive. In order to support fast paced decisions and enable strategic change at speed, it’s important to have data to support and drive those key decisions that deliver focused and targeted Campaigns. There are also some customers that perform Marketing efforts at levels of their customer segments both inside and outside of Marketo Engage. To support these different efforts, Marketo has created the ability to acquire large volumes of data in near real-time thru Data Streams.

Aside from the benefit of near real-time data, there are product-related benefits:

Data Streams are available to those that have purchased a Marketo Engage Performance Tier Package.

Lead Activity Data Stream Overview

Lead Activity Data Stream provides near real-time streaming of audit tracking Lead Activities where large volumes of Lead Activities can be sent to a customer’s external system. Streams enable customers to effectively audit Lead related events, usage patterns, provide views into Lead changes and trigger processes and workflows based upon the different types of Lead Events. There are over 144 Activity Types that can be subscribed to and sent thru the stream.

Types of Lead Data Streamed:

By providing views into Lead changes, this allows customers to make faster decisions on their overall marketing strategies and create more focused targeted campaigns. Some popular use cases would be:

List of Streamed Activities:

Note that if custom objects should be streamed, it must be all of the Lead-related custom objects. There is no way at the present time to select which ones are desired.

User Audit Data Stream Overview

User Audit Data Stream provides near real-time audit tracking of asset changes by users​. This enables a customer to effectively audit asset events, provide a view into user changes, and trigger processes or workflows based upon different types of audit events. Near real-time asset changes are sent via Adobe I/O events to a configurable endpoint. Audit events are broken down by Asset type and can subscribe to audit events that are important to them.

A good use case for subscribing to this stream would be:

List of Streamed User Audit Events:

Example of User Audit Event:

{
    "event_id": "a1b2c3d4-zyxw-9876-9z8y-a1b2c3d4e5f6",
    "event": {
        "specversion": "1.0",
        "id": "b77c743a-8e28-40f2-8aab-9541bbc85e68",
        "type": "com.adobe.platform.marketo.audit.user.email",
        "source": "https://www.marketo.com",
        "time": "2020-05-28T19:20:47.28Z",
        "datacontenttype": "application/json",
        "dataschema": "V1.0",
        "data": {
            "componentId": 232459,
            "componentType": "Email",
            "eventAction": "approve",
            "munchkinId": "123-ABC-456",
            "imsOrgId": "ADOBEORGID@AdobeOrg",
            "user": 253,
            "userId": "example@marketo.com"
        }
    }
}

Notification Data Stream Overview

Notification Data Stream is available as part of the Performance level offerings of Marketo Engage.

Currently, the notification center in Marketo can be configured to send notifications to an email address. Notification Data Stream enables the notifications to be sent directly to a configurable endpoint via Adobe I/O events. Notifications are provided thru the UI today and can be referenced by the orange bell in the top right of the screen and this stream takes those notifications and sends them thru a stream.

List of Notification Events:

Example of Notification Event:

{
    "event_id": "a1b2c3d4-zyxw-9876-9z8y-a1b2c3d4e5f6",
    "event": {
        "specversion": "1.0",
        "type": "com.adobe.platform.marketo.notification.campaign_abort",
        "source": "https://www.marketo.com",
        "time": "2021-05-27T10:22:37.489-5:00",
        "datacontenttype": "application/json",
        "dataschema": "V1.0",
        "data": {
            "componentType": "campaign_abort",
            "subType": "user_campaign_abort",
            "eventAction": {
                "campaignId":1234,
                "userId":"example@marketo.com",
            }
            "imsOrgId":"ADOBEORGID@AdobeOrg",
            "munchkinId":"123-ABC-456"
        }
    }
}

Technical Details

This section provides guidelines on what is needed, how to connect and receive streaming data for each of the streams. There is some level of coding and setup involved for each.

Lead Activity Data Stream

The Lead Activity Stream provides near real-time streaming of Marketo Lead Activity events and sends subscribed activity type changes with configurable attributes:

To implement the Lead Activity Data Stream, here are the steps for customers to follow:

Adobe then enables the data stream, at which point customers begin to receive data.

UML diagram of a typical Lead Activity Data Stream call:

Example of URL Endpoint Creation:

/*
Copyright 2022 Adobe
All Rights Reserved.

NOTICE: Adobe permits you to use, modify, and distribute this file in
accordance with the terms of the Adobe license agreement accompanying
it.
*/
constexpress=require('express')
constwinston=require('winston');
constport=3000

constapp=express().use(express.json())

constlogger=winston.createLogger({
  level: 'info',
  format: winston.format.json(),
  defaultMeta: {service: 'activity-stream-consumer-example'},
  transports: [
    // - Write all logs with level `error` and below to `error.log`
    newwinston.transports.File({filename: 'error.log',level: 'error'}),
    // - Write all logs with level `info` and below to `combined.log`
    newwinston.transports.File({filename: 'combined.log'}),
    newwinston.transports.Console({format: winston.format.simple()})
  ],
});

app.get('/',(req,res)=>{
  logger.info(JSON.stringify(req.query))
  res.sendStatus(200)
})

app.post('/',(req,res)=>{
  logger.info(JSON.stringify(req.body))
  res.sendStatus(200)
})

app.listen(port,()=>{
  logger.info(`app listening on port ${port}`)
})

A code sample for an application that consumes the Marketo Lead Activity Data Stream can be found here.

User Audit Data Stream and Notification Data Stream

User Audit events are sent to Adobe IO and can be consumed by logging in with an Adobe ID. Here are the steps to follow:

To setup Adobe IO, see Setting up Marketo User Audit Data Streams with Adobe IO in the Public Documentation section.

Setting Up The User Audit Data Stream in Marketo

The User Audit Data Stream is currently available as part of the Performance packages along with the other 3 Data Streams. For more information on the Packages, refer to the Product Description Page for Product limits and features.

Setting up Adobe I/O

See Getting Started with Adobe I/O Events

For basic instructions for this use case, starting from console.adobe.io:

When prompted, select either Create New Project or Add Event.

Get started with your new project

To start using Adobe services, add an API, events or runtime, view our documentation.

Public Documentation