Filter Catalog data using query parameters
The Catalog Service API allows response data to be filtered through the use of request query parameters. Part of best practices for Catalog is to use filters in all API calls, as they reduce the load on the API and help to improve overall performance.
This document outlines the most common methods for filtering Catalog objects in the API. It is recommended that you reference this document while reading the Catalog developer guide to learn more about how to interact with the Catalog API. For more general information on Catalog Service, see the Catalog overview.
Limit returned objects
The limit
query parameter constrains the number of objects returned in a response. Catalog responses are automatically metered according to configured limits:
- If a
limit
parameter is not specified, the maximum number of objects per response payload is 20. - For dataset queries, if
observableSchema
is requested using theproperties
query parameter, the maximum number of datasets returned is 20. - The global limit for all other Catalog queries is 100 objects.
- Invalid
limit
parameters (includinglimit=0
) result in 400-level error responses that outline proper ranges. - Limits or offsets that are passed as query parameters take precedence over those that are passed as headers.
API format
GET /{OBJECT_TYPE}?limit={LIMIT}
{OBJECT_TYPE}
The type of Catalog object to be retrieved. Valid objects are:
batches
dataSets
dataSetFiles
{LIMIT}
Request
The following request retrieves a list of datasets while limiting the response to three objects.
curl -X GET \
https://platform.adobe.io/data/foundation/catalog/dataSets?limit=3 \
-H 'Authorization: Bearer {ACCESS_TOKEN}' \
-H 'x-api-key: {API_KEY}' \
-H 'x-gw-ims-org-id: {ORG_ID}' \
-H 'x-sandbox-name: {SANDBOX_NAME}'
Response
A successful response returns a list of datasets, limited to the number indicated by the limit
query parameter.
{
"5ba9452f7de80400007fc52a": {
"name": "Sample Dataset 1",
"description": "Description of dataset.",
"files": "@/dataSetFiles?dataSetId=5ba9452f7de80400007fc52a"
},
"5bb276b03a14440000971552": {
"name": "Sample Dataset 2",
"description": "Description of dataset.",
"files": "@/dataSetFiles?dataSetId=5bb276b03a14440000971552"
},
"5bceaa4c26c115000039b24b": {
"name": "Sample Dataset 3"
}
}
Limit displayed properties
Even when filtering the number of objects returned using the limit
parameter, the returned objects themselves can often contain more information than you actually need. To further reduce the load on the system, it is best practice to filter responses to include only the properties that you need.
The properties
parameter filters response objects to return only a set of specified properties. The parameter can be set to return one or multiple properties.
The properties
parameter can accept any level object properties. sampleKey
can be extracted using ?properties=subItem.sampleKey
.
{
"5ba9452f7de80400007fc52a": {
"name": "Sample Dataset",
"description": "Sample dataset containing important data",
"subitem": {
"sampleKey": "sampleValue"
}
}
}
API format
GET /{OBJECT_TYPE}?properties={PROPERTY}
GET /{OBJECT_TYPE}?properties={PROPERTY_1},{PROPERTY_2},{PROPERTY_3}
GET /{OBJECT_TYPE}/{OBJECT_ID}?properties={PROPERTY_1},{PROPERTY_2},{PROPERTY_3}
{OBJECT_TYPE}
The type of Catalog object to be retrieved. Valid objects are:
batches
dataSets
dataSetFiles
{PROPERTY}
{OBJECT_ID}
Request
The following request retrieves a list of datasets. The comma-separated list of property names provided under the properties
parameter indicates the properties to be returned in the response. A limit
parameter is also included, which limits the number of datasets returned. If the request did not include a limit
parameter, the response would contain a maximum of 20 objects.
curl -X GET \
'https://platform.adobe.io/data/foundation/catalog/dataSets?limit=4&properties=name,schemaRef' \
-H 'Authorization: Bearer {ACCESS_TOKEN}' \
-H 'x-api-key: {API_KEY}' \
-H 'x-gw-ims-org-id: {ORG_ID}' \
-H 'x-sandbox-name: {SANDBOX_NAME}'
Response
A successful response returns a list of Catalog objects with only the requested properties displayed.
{
"Dataset1": {
"name": "Dataset 1",
"schemaRef": {
"id": "https://ns.adobe.com/{TENANT_ID}/schemas/bc82c518380478b59a95c63e0f843121",
"contentType": "application/vnd.adobe.xed+json;version=1"
}
},
"Dataset2": {},
"Dataset3": {
"name": {},
},
"Dataset4": {
"name": "Dataset 4",
"schemaRef": {
"id": "https://ns.adobe.com/{TENANT_ID}/schemas/142afb78d8b368a5ba97a6cc8fc7e033",
"contentType": "application/vnd.adobe.xed+json;version=1"
}
}
}
Based on the response above, the following can be inferred:
- If an object is missing any requested properties, it will only show the requested properties that it does include. (
Dataset1
) - If an object does not include any of the requested properties, it will appear as an empty object. (
Dataset2
) - A dataset may return a requested property as an empty object if it contains the property but there is no value. (
Dataset3
) - Otherwise, the dataset will display the full value of all requested properties. (
Dataset4
)
schemaRef
property for each dataset, the version number indicates the latest minor version of the schema. See the section on schema versioning in the XDM API guide for more information.Offset starting index of response list
The start
query parameter offsets the response list forward by a specified number, using zero-based numbering. For example, start=2
would offset the response to start on the third listed object.
If the start
parameter is not paired with a limit
parameter, the maximum number of objects returned is 20.
API format
GET /{OBJECT_TYPE}?start={OFFSET}
{OBJECT_TYPE}
The type of Catalog object to be retrieved. Valid objects are:
batches
dataSets
dataSetFiles
{OFFSET}
Request
The following request retrieves a list of datasets, offsetting to the fifth object (start=4
) and limiting the response to two returned datasets (limit=2
).
curl -X GET \
https://platform.adobe.io/data/foundation/catalog/dataSets?start=4&limit=2 \
-H 'Authorization: Bearer {ACCESS_TOKEN}' \
-H 'x-api-key: {API_KEY}' \
-H 'x-gw-ims-org-id: {ORG_ID}' \
-H 'x-sandbox-name: {SANDBOX_NAME}'
Response
The response includes a JSON object containing two top-level items (limit=2
), one for each dataset and their details (details have been condensed in the example). The response is shifted by four (start=4
), meaning the datasets shown are number five and six chronologically.
{
"Dataset5": {},
"Dataset6": {}
}
Filter by tag
Some Catalog objects support the use of a tags
attribute. Tags can attach information to an object and then be used later to retrieve that object. The choice of what tags to use and how to apply them depends on your organizational processes.
There are a few limitations to consider when using tags:
-
The only Catalog objects that currently support tags are datasets, batches, and connections.
-
Tag names are unique to your organization.
-
Adobe processes may leverage tags for certain behaviors. The names of these tags are prefixed with “adobe” as a standard. Therefore, you should avoid this convention when declaring tag names.
-
The following tag names are reserved for use across Experience Platform, and therefore cannot be declared as a tag name for your organization:
unifiedProfile
: This tag name is reserved for datasets to be ingested by Real-Time Customer Profile.unifiedIdentity
: This tag name is reserved for datasets to be ingested by Identity Service.
Below is an example of a dataset that contains a tags
property. The tags within that property take the form of key-value pairs, with each tag value appearing as an array containing a single string:
{
"5be1f2ecc73c1714ceba66e2": {
"imsOrg": "{ORG_ID}",
"tags": {
"sampleTag": [
"123456"
],
"secondTag": [
"sample_tag_value"
]
},
"name": "Sample Dataset",
"description": "Same dataset containing sample data.",
"createdUser": "{CREATED_USER}",
"createdClient": "{CREATED_CLIENT}",
"updatedUser": "{UPDATED_USER}",
"version": "1.0.0",
"created": 1541534444286,
"updated": 1541534444286
}
}
API format
Values for the tags
parameter take the form of key-value pairs, using the format {TAG_NAME}:{TAG_VALUE}
. Multiple key-value pairs can be provided in the form of a comma-separated list. When multiple tags are provided, an AND relationship is assumed.
The parameter supports wildcard characters (*
) for tag values. For example, a search string of test*
returns any object where the tag value begins with “test”. A search string consisting solely of a wildcard can be used to filter objects based on whether or not they contain a specific tag, regardless of its value.
GET /{OBJECT_TYPE}?tags={TAG_NAME}:{TAG_VALUE}
GET /{OBJECT_TYPE}?tags={TAG_NAME_1}:{TAG_VALUE_1},{TAG_NAME_2}:{TAG_VALUE_2}
GET /{OBJECT_TYPE}?tags={TAG_NAME}:{TAG_VALUE}*
GET /{OBJECT_TYPE}?tags={TAG_NAME}:*
{OBJECT_TYPE}
The type of Catalog object to be retrieved. Valid objects are:
batches
dataSets
{TAG_NAME}
{TAG_VALUE}
*
).Request
The following request retrieves a list of datasets, filtering by one tag having a specific value AND the second tag being present.
curl -X GET \
'https://platform.adobe.io/data/foundation/catalog/dataSets?tags=sampleTag:123456,secondTag:* \
-H 'Authorization: Bearer {ACCESS_TOKEN}' \
-H 'x-api-key: {API_KEY}' \
-H 'x-gw-ims-org-id: {ORG_ID}' \
-H 'x-sandbox-name: {SANDBOX_NAME}'
Response
A successful response returns a list of datasets that contain sampleTag
with a value of “123456”, AND secondTag
with any value. Unless a limit is also specified, the response contains a maximum of 20 objects.
{
"5b67f4dd9f6e710000ea9da4": {
"version": "1.0.2",
"imsOrg": "{ORG_ID}",
"name": "Example Dataset 1",
"created": 1533539550237,
"updated": 1533539552416,
"createdClient": "{API_KEY}",
"createdUser": "{USER_ID}",
"updatedUser": "{USER_ID}",
"tags": {
"sampleTag": [
"123456"
],
"secondTag": [
"Example tag value"
]
},
},
"5b1e3c867e6d2600003d5b49": {
"version": "1.0.0",
"imsOrg": "{ORG_ID}",
"name": "Example Dataset 2",
"created": 1533539550237,
"updated": 1533539552416,
"createdClient": "{API_KEY}",
"createdUser": "{USER_ID}",
"updatedUser": "{USER_ID}",
"tags": {
"sampleTag": [
"123456"
],
"secondTag": [
"A different tag value"
],
"anotherTag": [
"2.0"
]
},
}
}
Filter by date range
Some endpoints in the Catalog API have query parameters that allow for ranged queries, most often in the case of dates.
API format
GET /batches?createdAfter={TIMESTAMP_1}&createdBefore={TIMESTAMP_2}
{TIMESTAMP }
Request
The following request retrieves a list of batches created during the month of April 2019.
curl -X GET \
'https://platform.adobe.io/data/foundation/catalog/batches?createdAfter=1554076800000&createdBefore=1556668799000' \
-H 'Authorization: Bearer {ACCESS_TOKEN}' \
-H 'x-api-key: {API_KEY}' \
-H 'x-gw-ims-org-id: {ORG_ID}' \
-H 'x-sandbox-name: {SANDBOX_NAME}'
Response
A successful response contains a list of Catalog objects that fall within the specified date range. Unless a limit is also specified, the response contains a maximum of 20 objects.
{
"5b67f4dd9f6e710000ea9da4": {
"version": "1.0.2",
"imsOrg": "{ORG_ID}",
"name": "Example Dataset 1",
"created": 1554930967705,
"updated": 1554931119718,
"createdClient": "{API_KEY}",
"createdUser": "{USER_ID}",
"updatedUser": "{USER_ID}",
},
"5b1e3c867e6d2600003d5b49": {
"version": "1.0.0",
"imsOrg": "{ORG_ID}",
"name": "Example Dataset 2",
"created": 1554974386247,
"updated": 1554974386268,
"createdClient": "{API_KEY}",
"createdUser": "{USER_ID}",
"updatedUser": "{USER_ID}",
}
}
Sort by property
The orderBy
query parameter allows you to sort (order) response data based on a specified property value. This parameter requires a “direction” (asc
for ascending or desc
for descending), followed by a colon (:
) and then a property to sort the results by. If a direction is not specified, the default direction is ascending.
Multiple sorting properties can be provided in a comma-separated list. If the first sorting property produces multiple objects that contain the same value for that property, the second sorting property is then used to further sort those matching objects.
For example, consider the following query: orderBy=name,desc:created
. Results are sorted in ascending order based on the first sorting property, name
. In cases where multiple records share the same name
property, those matching records are then sorted by the second sorting property, created
. If no returned records share the same name
, the created
property does not factor into the sorting.
API format
GET /{OBJECT_TYPE}?orderBy=asc:{PROPERTY_NAME}
GET /{OBJECT_TYPE}?orderBy=desc:{PROPERTY_NAME}
GET /{OBJECT_TYPE}?orderBy={PROPERTY_NAME_1},desc:{PROPERTY_NAME_2}
{OBJECT_TYPE}
The type of Catalog object to be retrieved. Valid objects are:
batches
dataSets
dataSetFiles
{PROPERTY_NAME}
Request
The following request retrieves a list of datasets sorted by their name
property. If any datasets share the same name
, those datasets will in turn be ordered by their updated
property in descending order.
curl -X GET \
'https://platform.adobe.io/data/foundation/catalog/dataSets?orderBy=name,desc:updated' \
-H 'Authorization: Bearer {ACCESS_TOKEN}' \
-H 'x-api-key: {API_KEY}' \
-H 'x-gw-ims-org-id: {ORG_ID}' \
-H 'x-sandbox-name: {SANDBOX_NAME}'
Response
A successful response contains a list of Catalog objects that are sorted according to the orderBy
parameter. Unless a limit is also specified, the response contains a maximum of 20 objects.
{
"5b67f4dd9f6e710000ea9da4": {
"version": "1.0.2",
"imsOrg": "{ORG_ID}",
"name": "0405",
"created": 1554930967705,
"updated": 1554931119718,
"createdClient": "{API_KEY}",
"createdUser": "{USER_ID}",
"updatedUser": "{USER_ID}",
},
"5b1e3c867e6d2600003d5b49": {
"version": "1.0.3",
"imsOrg": "{ORG_ID}",
"name": "AAM Dataset",
"created": 1554974386247,
"updated": 1554974386268,
"createdClient": "{API_KEY}",
"createdUser": "{USER_ID}",
"updatedUser": "{USER_ID}",
},
"5cd3a129ec106214b722a939": {
"version": "1.0.2",
"imsOrg": "{ORG_ID}",
"name": "AAM Dataset",
"created": 1554028394852,
"updated": 1554130582960,
"createdClient": "{API_KEY}",
"createdUser": "{USER_ID}",
"updatedUser": "{USER_ID}",
}
}
Filter by property
Catalog provides two methods of filtering by property, which are further outlined in the sections that follow:
- Using simple filters: Filter by whether a specific property matches a specific value.
- Using the property parameter: Use conditional expressions to filter based whether a property exists, or if a property’s value matches, approximates, or compares to another specified value or regular expression.
Using simple filters using-simple-filters
Simple filters allow you to filter responses based on specific property values. A simple filter takes the form of {PROPERTY_NAME}={VALUE}
.
For example, the query name=exampleName
returns only objects whose name
property contains a value of “exampleName”. By contrast, the query name=!exampleName
returns only objects whose name
property is not “exampleName”.
In addition, simple filters support the ability to query for multiple values for a single property. When multiple values are provided, the response returns objects whose property matches any of the values in the provided list. You can invert a multi-value query by prefixing a !
character to the list, returning only objects whose property value is not in the provided list (for example, name=!exampleName,anotherName
).
API format
GET /{OBJECT_TYPE}?{PROPERTY_NAME}={VALUE}
GET /{OBJECT_TYPE}?{PROPERTY_NAME}=!{VALUE}
GET /{OBJECT_TYPE}?{PROPERTY_NAME}={VALUE_1},{VALUE_2},{VALUE_3}
GET /{OBJECT_TYPE}?{PROPERTY_NAME}=!{VALUE_1},{VALUE_2},{VALUE_3}
{OBJECT_TYPE}
The type of Catalog object to be retrieved. Valid objects are:
batches
dataSets
dataSetFiles
{PROPERTY_NAME}
{VALUE}
Request
The following request retrieves a list of datasets, filtered to include only datasets whose name
property has a value of “exampleName” or “anotherName”.
curl -X GET \
'https://platform.adobe.io/data/foundation/catalog/dataSets?name=exampleName,anotherName' \
-H 'Authorization: Bearer {ACCESS_TOKEN}' \
-H 'x-api-key: {API_KEY}' \
-H 'x-gw-ims-org-id: {ORG_ID}' \
-H 'x-sandbox-name: {SANDBOX_NAME}'
Response
A successful response contains a list of datasets, excluding any datasets whose name
is “exampleName” or “anotherName”. Unless a limit is also specified, the response contains a maximum of 20 objects.
{
"5b67f4dd9f6e710000ea9da4": {
"version": "1.0.2",
"imsOrg": "{ORG_ID}",
"name": "exampleName",
"created": 1554930967705,
"updated": 1554931119718,
"createdClient": "{API_KEY}",
"createdUser": "{USER_ID}",
"updatedUser": "{USER_ID}",
},
"5b1e3c867e6d2600003d5b49": {
"version": "1.0.3",
"imsOrg": "{ORG_ID}",
"name": "anotherName",
"created": 1554974386247,
"updated": 1554974386268,
"createdClient": "{API_KEY}",
"createdUser": "{USER_ID}",
"updatedUser": "{USER_ID}",
}
}
Using the property
parameter using-the-property-parameter
The property
query parameter provides more flexibility for property-based filtering than simple filters. In addition to filtering based on whether a property has a specific value, the property
parameter can use other comparison operators (such as “more-than” (>
) and “less-than” (<
)) as well as regular expressions to filter by property values. It can also filter by whether or not a property exists, regardless of its value.
The property
parameter can accept any level object properties. sampleKey
can be used for filtering using ?properties=subItem.sampleKey
.
{
"5ba9452f7de80400007fc52a": {
"name": "Sample Dataset",
"description": "Sample dataset containing important data",
"subitem": {
"sampleKey": "sampleValue"
}
}
}
API format
GET /{OBJECT_TYPE}?property={CONDITION}
{OBJECT_TYPE}
The type of Catalog object to be retrieved. Valid objects are:
batches
dataSets
dataSetFiles
{CONDITION}
The value of the property
parameter supports several different kinds of conditional expressions. The following table outlines the basic syntax for supported expressions:
property=name
!
” to the value of a property
parameter returns only objects where the property does not exist.property=!name
~
) symbol.property=name~^example
==
).property=name==exampleName
!=
).property=name!=exampleName
property=version<1.0.0
property=version<=1.0.0
property=version>1.0.0
property=version>=1.0.0
name
property supports the use of a wildcard *
, either as the entire search string or as a part of it. Wildcards match empty characters, such that the search string te*st
will match the value “test”. Asterisks are escaped by doubling them (**
). A double-asterisk in a search string represents a single asterisk as a literal string.Request
The following request will return any datasets with a version number greater than 1.0.3.
curl -X GET \
https://platform.adobe.io/data/foundation/catalog/dataSets?property=version>1.0.3 \
-H 'Authorization: Bearer {ACCESS_TOKEN}' \
-H 'x-api-key: {API_KEY}' \
-H 'x-gw-ims-org-id: {ORG_ID}' \
-H 'x-sandbox-name: {SANDBOX_NAME}'
Response
A successful response contains a list of datasets whose version numbers are greater than 1.0.3. Unless a limit is also specified, the response contains a maximum of 20 objects.
{
"5b67f4dd9f6e710000ea9da4": {
"version": "1.1.2",
"imsOrg": "{ORG_ID}",
"name": "sampleDataset",
"created": 1554930967705,
"updated": 1554931119718,
"createdClient": "{API_KEY}",
"createdUser": "{USER_ID}",
"updatedUser": "{USER_ID}",
},
"5b1e3c867e6d2600003d5b49": {
"version": "1.0.6",
"imsOrg": "{ORG_ID}",
"name": "exampleDataset",
"created": 1554974386247,
"updated": 1554974386268,
"createdClient": "{API_KEY}",
"createdUser": "{USER_ID}",
"updatedUser": "{USER_ID}",
},
"5cd3a129ec106214b722a939": {
"version": "1.0.4",
"imsOrg": "{ORG_ID}",
"name": "anotherDataset",
"created": 1554028394852,
"updated": 1554130582960,
"createdClient": "{API_KEY}",
"createdUser": "{USER_ID}",
"updatedUser": "{USER_ID}",
}
}
Combine multiple filters
Using an ampersand (&
), you can combine multiple filters in a single request. When additional conditions are added to a request, an AND relationship is assumed.
API format
GET /{OBJECT_TYPE}?{FILTER_1}={VALUE}&{FILTER_2}={VALUE}&{FILTER_3}={VALUE}