Create an export workflow (Part 2): Extract, format, and save data to an external account

In this video, we’re going to continue creating the Adobe Experience Platform Campaign Logs Export Workflow. In a previous video, we started a workflow by adding a JavaScript component and query component. For this tutorial, we will create the second half of the workflow, which involves formatting our data, uploading the four minute data to an external account and saving the date for our next run.

To begin, let’s select the actions tab, then drag and drop data extraction to our workflow, connecting it to our query.

Upon double clicking, the data extraction pop-over appears. Let’s start by providing a new label. Since we are using the data extraction to format and save the file, let’s call this format and save file.

Next, we need to add a file name. I have added a timestamp to the file name, to help track the data.

Later in this course, we need to connect Platform using a source to the S3 bucket we are exporting our files to. A source and Platform works by monitoring a specific bucket and path. Based on the sources schedule, every time a source runs, it looks at all the files in our directory and tracks what files have already been processed and only processes the new files.

Once we have our file name, select edit the file format to open the data to extract pop-over. We need to define what output columns we want to extract from our list that was queried. It is important to note that this step is closely related to defining a schema in Platform. This means that if you already have a schema you want to use in Platform to store your data, the output columns you select need to match your Platform schema. In this example, I’m going to add output values that I plan on using when I create my Platform schema later in this course.

After adding all the output columns, we can select next, or you can choose to update the labels. Additionally, if you have a CRM ID, for example, it is strongly recommended you use that ID as a primary identifier. That being said, it is not necessary to rename the labels in Campaign because when we eventually connect to the Platform using sources, we will have the opportunity to map our output columns to the schema columns used in Platform. Select next until you reach the destination format page. Here, we need to change a couple of values starting with the output format. From the dropdown, select CSV. Set the encoding to be UTF-8, then select edit for the date format and change the date separator to a dash. Then tick the display milliseconds box. We are changing the format to match the optimal format used by experience Platform. Once complete select, okay, followed by selecting finish then selecting okay one last time. Now that we are formatting and saving our data, we need to specify where this data should be stored. Within the events tab, drag and drop the file transfer component and attach it to the end of our format and save component. Double click the component and the file transfer pop-over appears.

Let’s start by changing the label to upload to external account. Next let’s change the action to file upload from the action drop-down. Finally, we need to select our external account. In this example, the account is called Platform Integration and the server folder I want to save my data to is Platform_data.

If you wanted to use a new server folder, you would just need to configure the folder in your external account.

Once complete, select okay.

The final step in our workflow is to add another JavaScript code component. Similar to the start of our workflow, copy and paste the save date JavaScript file found in the assets section into the script field. This JavaScript code adds next runtime to the database so that we know when the starting point for the next run should occur. This prevents us from re uploading data that is already present in our S3 bucket. In other words, this script tells our workflow that anything from the time this was last executed to the time we execute the workflow next should be queried and processed.

Now that we have all our components, we can simply select the flow control tab, then drag and drop end to the workflow. I’m also going to quickly rename the label to end with success in brackets.

As an additional best practice, let’s drop a second end component and rename this one error. Sometimes it can be difficult to diagnose a problem when uploading, formatting or saving data. By adding an error row, this becomes easier to troubleshoot. Double-click our JavaScript code component. Then at the bottom, select the process errors tick box followed by selecting okay. This will create a branch in our workflow that we can connect to our error component. After doing the same for the format and save file, as well as the save date for next run components, our workflow should look similar to what’s on the screen.

You should now have a workflow that exports our Adobe Campaign Logs to an external account. Thanks for watching. -