Create a Marketo Engage source connection and dataflow in the UI
This tutorial provides steps for creating a Marketo Engage (hereinafter referred to as “Marketo”) source connector in the UI to bring B2B data into Adobe Experience Platform.
Get started
This tutorial requires a working understanding of the following components of Adobe Experience Platform:
- B2B namespaces and schema auto-generation utility: The B2B namespaces and schema auto-generation utility allows you to use Postman to auto-generate values for your B2B namespaces and schemas. You must complete your B2B namespaces and schemas first, before creating a Marketo source connection and dataflow.
- Sources: Experience Platform allows data to be ingested from various sources while providing you with the ability to structure, label, and enhance incoming data using Platform services.
- Experience Data Model (XDM): The standardized framework by which Experience Platform organizes customer experience data.
- Create and edit schemas in the UI: Learn how to create and edit schemas in the UI.
- Identity namespaces: Identity namespaces are a component of Identity Service that serve as indicators of the context to which an identity relates. A fully qualified identity includes an ID value and a namespace.
- Real-Time Customer Profile: Provides a unified, real-time consumer profile based on aggregated data from multiple sources.
- Sandboxes: Experience Platform provides virtual sandboxes which partition a single Platform instance into separate virtual environments to help develop and evolve digital experience applications.
Gather required credentials
In order to access your Marketo account on Experience Platform, you must provide the following values:
munchkinIdclientIdclientSecretFor more information on acquiring these values, refer to the Marketo authentication guide.
Once you have gathered your required credentials, you can follow the steps in the next section.
Connect your Marketo account
In the Platform UI, select Sources from the left navigation to access the Sources workspace. You can select the appropriate category from the catalog on the left-hand side of your screen. Alternatively, you can find the specific source you wish to work with using the search option.
Under the Adobe applications category, select Marketo Engage, and then select Add data.
The Connect Marketo Engage account page appears. On this page, you can either use a new account or access an existing account.
To create a new account, select New account and provide a name, an optional description, and your credentials.
When finished, select Connect to source and then allow some time for the new connection to establish.
To use an existing account, select Existing account and then select the account that you want to use from the existing account catalog.
Select Next to proceed.
Select a dataset
After creating your Marketo account, the next step provides an interface for you to explore Marketo datasets.
The left half of the interface is a directory browser, displaying the 10 Marketo datasets. A fully-functioning Marketo source connection requires the ingestion of the nine different datasets. If you are also using the Marketo account-based marketing (ABM) feature, then you must also create a 10th dataflow to ingest the Named Accounts dataset.
Select the dataset that you want to ingest. This updates the interface to display a preview of your dataset. When finished, select Next.
Provide dataset and dataflow details provide-dataset-and-dataflow-details
Next, you must provide information on your dataset and your dataflow.
Dataset details dataset-details
A dataset is a storage and management construct for a collection of data, typically a table, that contains a schema (columns) and fields (rows). Data that is successfully ingested into Experience Platform is stored within the data lake as datasets. During this step, you can create a new dataset or use an existing dataset.
To use a new dataset, select New dataset and then provide a name, and an optional description for your dataset. You must also select an Experience Data Model (XDM) schema that your dataset adheres to.
If you already have an existing dataset, select Existing dataset and then use the Advanced search option to view a window of all datasets in your organization, including their respective details, such as whether they are enabled for ingestion to Real-Time Customer Profile or not.
Dataflow configurations dataflow-configurations
If your dataset is enabled for Real-Time Customer Profile, then during this step, you can toggle Profile dataset to enable your data for Profile-ingestion. You can also use this step to enable Error diagnostics and Partial ingestion.
- Error diagnostics: Select Error diagnostics to instruct the source to produce error diagnostics that you can later reference when monitoring your dataset activity and dataflow status.
- Partial ingestion: Partial batch ingestion is the ability to ingest data containing errors, up to a certain configurable threshold. This feature allows you to successfully ingest all of your accurate data into Experience Platform, while all of your incorrect data is batched separately with information on why it is invalid.
During this step, you can enable Sample dataflow to limit data ingestion and avoid additional costs that come with ingesting all historical data, including Person identities.
Quick guide on using sample dataflow
Sample dataflow is a configuration that you can set for your Marketo dataflow to limit your ingestion rate and then try out Experience Platform features without having to ingest large amounts of data.
- Enable sample dataflow to limit historical data by ingesting up to 100k (from the largest record ID) records or up to the last 10 days of activity during the backfill job.
- When using the sample dataflow configuration for all B2B entities, you must consider that it is possible that some related records may be missing because the entire history of the source data does not get ingested.
Additionally, if you are ingesting data from the companies dataset, you can enable Exclude unclaimed accounts to exclude unclaimed accounts from ingestion.
When individuals fill out a form, Marketo creates a phantom account record based on the Company Name that contains no other data. For new dataflows, the toggle to exclude unclaimed accounts is enabled by default. For existing dataflows, you can enable or disable the feature, with changes applying to newly ingested data and not existing data.
Map your Marketo dataset source fields to target XDM fields
The Mapping step appears, providing you with an interface to map the source fields from your source schema to their appropriate target XDM fields in the target schema.
Each Marketo dataset has its own specific mapping rules to follow. See the following for more information on how to map Marketo datasets to XDM:
Based on your needs, you can choose to map fields directly, or use data prep functions to transform source data to derive computed or calculated values. For comprehensive steps on using the mapping interface, see the Data Prep UI guide.
Once your mapping sets are ready, select Next and allow for a few moments for the new dataflow to be created.
Review your dataflow
The Review step appears, allowing you to review your new dataflow before it is created. Details are grouped within the following categories:
- Connection: Shows the source type, the relevant path of the chosen source entity, and the amount of columns within that source entity.
- Assign dataset & map fields: Shows which dataset the source data is being ingested into, including the schema that the dataset adheres to.
Once you have reviewed your dataflow, select Save & ingest and allow some time for the dataflow to be created.
Monitor your dataflow
Once your dataflow has been created, you can monitor the data that is being ingested through it to see information on ingestion rates, success, and errors. For more information on how to monitor dataflows, see the tutorial on monitoring dataflows in the UI.
Delete your attributes
Custom attributes in datasets cannot be retroactively hidden or removed. If you want to hide or remove a custom attribute from an existing dataset, then you must create a new dataset without this custom attribute, a new XDM schema, and configure a new dataflow for the new dataset that you create. You must also disable or delete the original dataflow that consists of the dataset with the custom attribute you want to hide or remove.
Delete your dataflow
You can delete dataflows that are no longer necessary or were incorrectly created using the Delete function available in the Dataflows workspace. For more information on how to delete dataflows, see the tutorial on deleting dataflows in the UI.
Next steps
By following this tutorial, you have successfully created a dataflow to ingest B2B data from your Marketo Engage source to Experience Platform.
Appendix appendix
The following sections provide additional guidelines that you may follow when using the Marketo source.
Error messages in the UI error-messages
The following error messages are displayed in the UI when Platform detects issues with your setup:
Munchkin ID is not mapped to the appropriate organization
Authentication will be denied if your Munchkin ID is not mapped to the Platform organization that you are using. Configure the mapping between your Munchkin ID and your organization using the Marketo interface.
Primary identity is missing
A dataflow will fail to save and ingest if a primary identity is missing. Ensure that a primary identity exists within your XDM schema, before attempting to configure a dataflow.
