Query Service and Data Distiller overview

Adobe Experience Platform Query Service is a set of tools that allows you to query the contents of Experience Platform datasets. To better understand how Query Service works in context, it’s helpful to know how data is ingested and stored in Platform at a high level. Platform has multiple ingestion patterns depending on where data is coming from, but in most cases the data is ingested into one or more datasets as columns in rows. These datasets are stored in the data lake, which is a centralized data store that’s populated to each Experience Platform sandbox. The data lake is one of Experience Platform’s key data stores where customer Experience data can be further processed and sent to downstream applications like Adobe Technologies or third-party systems. Experience Platform’s other key data stores are the Identity Graph and Profile Store, which are separate from the data lake. These are where Experience Platform stores and manages customer identities and profiles respectively. Why is all this important for Query Service? Well, everything that you can do in Query Service revolves around what lives in the data lake specifically. You can’t use it to query identity graphs or profiles since those are accessed using other tools like segmentation and the Identity Graph Viewer. If your organization has purchased the Data Distiller add-on package for an Experience Platform application, you can go beyond read-only operations and use Query Service to actively transform and insert data into the data lake. As a foundational core service in Experience Platform, Query Service is included in all platform-based applications, specifically Adobe Realtime Customer Data Platform, Adobe Journey Optimizer, Adobe Customer Journey Analytics, and Adobe Mix Modeler. Since Experience Platform is extremely flexible when it comes to the range of sources and structures of data it can ingest, Query Service is equally flexible in the kinds of questions about that data. However, your access to specific Query Service capabilities may vary depending on which products and add-ons your organization has purchased. Let’s go over how this breaks down and introduce you to the service’s core features while we’re at it. In the platform interface, you can access Query Service features by selecting Queries in the left navigation. From the landing page, select Create Query and you’ll be brought to the Query Editor. Using the Editor, you can write custom queries for your platform datasets using standard SQL. For example, writing a simple SELECT query, you can see I provide the name of a platform dataset as part of the FROM clause, indicating which dataset I want to query. This is called the table name of the dataset, which you can find by going to the Datasets tab and opening the details of the dataset in question. To run this query, I’ll select the Play icon, and you can see that the results of the query are immediately output in the area below. By executing ad hoc queries like this, you can quickly explore and validate customer experience data. In addition to standard SQL syntax, there are also special Adobe-defined functions that help you more flexibly access and organize experience data. Using these functions, you can group related events into sessions, leverage pathing contexts like the previous page view for a given event, analyze time gaps between specific event types, Query service also provides authentication credentials, letting you execute queries from external clients if preferred. Any platform-based application user can execute queries like this, provided they’ve been granted Manage Queries permission from an administrator. At any given time, your organization’s users can execute a maximum of four queries concurrently. You can also use queries to write results back to another dataset for further actioning and analysis. By outputting query results to datasets, analysts and data engineers can clean, transform, and enrich customer experience data in addition to exploring and validating through ad hoc queries. Queries that write their results to datasets can also be scheduled and centrally managed as batch jobs. To use these queries, your organization must have previously purchased the Data Distiller add-on package for a platform-based application. With the Data Distiller add-on, you’ll see additional controls such as the Scheduled Queries tab and the Create Table as Select option within the Query Editor interface. So that was a brief overview of Query Service and its core capabilities, including which capabilities are available to all users of platform-based applications versus those that are exclusive to the Data Distiller add-on package. We only scratch the surface when it comes to the many ways Query Service can be used, so we strongly encourage you to experiment with your own queries and to check out our tutorials and technical documentation to learn more about different use cases and processes. Thanks for watching!