Filter Catalog data using query parameters

To improve performance and retrieve relevant results, use query parameters to filter Catalog Service API response data.

Learn about common filtering methods for Catalog objects in the API. Use this document alongside the Catalog Developer Guide for more details on API interactions. For general Catalog Service information, see the Catalog overview.

Limit returned objects limit-returned-objects

The limit query parameter restricts the number of objects returned in a response. Catalog responses follow predefined limits:

API format

GET /{OBJECT_TYPE}?limit={LIMIT}

Request

The following request retrieves a list of datasets while limiting the response to three objects.

curl -X GET \
  https://platform.adobe.io/data/foundation/catalog/dataSets?limit=3 \
  -H 'Authorization: Bearer {ACCESS_TOKEN}' \
  -H 'x-api-key: {API_KEY}' \
  -H 'x-gw-ims-org-id: {ORG_ID}' \
  -H 'x-sandbox-name: {SANDBOX_NAME}'

Response

A successful response returns a list of datasets, limited to the number indicated by the limit query parameter.

{
    "5ba9452f7de80400007fc52a": {
        "name": "Sample Dataset 1",
        "description": "Description of dataset.",
        "files": "@/dataSetFiles?dataSetId=5ba9452f7de80400007fc52a"
    },
    "5bb276b03a14440000971552": {
        "name": "Sample Dataset 2",
        "description": "Description of dataset.",
        "files": "@/dataSetFiles?dataSetId=5bb276b03a14440000971552"
    },
    "5bceaa4c26c115000039b24b": {
        "name": "Sample Dataset 3"
    }
}

Limit displayed properties limit-displayed-properties

Even when filtering the number of objects returned using the limit parameter, the returned objects themselves can often contain more information than you actually need. To further reduce the load on the system, it is best practice to filter responses to include only the properties that you need.

The properties parameter filters response objects to return only a set of specified properties. The parameter can be set to return one or multiple properties.

The properties parameter can accept any level object properties. sampleKey can be extracted using ?properties=subItem.sampleKey.

{
  "5ba9452f7de80400007fc52a": {
      "name": "Sample Dataset",
      "description": "Sample dataset containing important data",
      "subitem": {
          "sampleKey": "sampleValue"
      }
  }
}

API format

GET /{OBJECT_TYPE}?properties={PROPERTY}
GET /{OBJECT_TYPE}?properties={PROPERTY_1},{PROPERTY_2},{PROPERTY_3}
GET /{OBJECT_TYPE}/{OBJECT_ID}?properties={PROPERTY_1},{PROPERTY_2},{PROPERTY_3}

Request

The following request retrieves a list of datasets. The comma-separated list of property names provided under the properties parameter indicates the properties to be returned in the response. A limit parameter is also included, which limits the number of datasets returned. If the request did not include a limit parameter, the response would contain a maximum of 20 objects.

curl -X GET \
  'https://platform.adobe.io/data/foundation/catalog/dataSets?limit=4&properties=name,schemaRef' \
  -H 'Authorization: Bearer {ACCESS_TOKEN}' \
  -H 'x-api-key: {API_KEY}' \
  -H 'x-gw-ims-org-id: {ORG_ID}' \
  -H 'x-sandbox-name: {SANDBOX_NAME}'

Response

A successful response returns a list of Catalog objects with only the requested properties displayed.

{
    "Dataset1": {
        "name": "Dataset 1",
        "schemaRef": {
            "id": "https://ns.adobe.com/{TENANT_ID}/schemas/bc82c518380478b59a95c63e0f843121",
            "contentType": "application/vnd.adobe.xed+json;version=1"
        }
    },
    "Dataset2": {},
    "Dataset3": {
        "name": {},
    },
    "Dataset4": {
        "name": "Dataset 4",
        "schemaRef": {
            "id": "https://ns.adobe.com/{TENANT_ID}/schemas/142afb78d8b368a5ba97a6cc8fc7e033",
            "contentType": "application/vnd.adobe.xed+json;version=1"
        }
    }
}

Based on the response above, the following can be inferred:

Offset starting index of response list offset-starting-index

The start query parameter offsets the response list forward by a specified number, using zero-based numbering. For example, start=2 would offset the response to start on the third listed object.

If the start parameter is not paired with a limit parameter, the maximum number of objects returned is 20.

API format

GET /{OBJECT_TYPE}?start={OFFSET}

Request

The following request retrieves a list of datasets, offsetting to the fifth object (start=4) and limiting the response to two returned datasets (limit=2).

curl -X GET \
  https://platform.adobe.io/data/foundation/catalog/dataSets?start=4&limit=2 \
  -H 'Authorization: Bearer {ACCESS_TOKEN}' \
  -H 'x-api-key: {API_KEY}' \
  -H 'x-gw-ims-org-id: {ORG_ID}' \
  -H 'x-sandbox-name: {SANDBOX_NAME}'

Response

The response includes a JSON object containing two top-level items (limit=2), one for each dataset and their details (details have been condensed in the example). The response is shifted by four (start=4), meaning the datasets shown are number five and six chronologically.

{
    "Dataset5": {},
    "Dataset6": {}
}

Filter by tag

Some Catalog objects support the use of a tags attribute. Tags can attach information to an object and then be used later to retrieve that object. The choice of what tags to use and how to apply them depends on your organizational processes.

There are a few limitations to consider when using tags:

Below is an example of a dataset that contains a tags property. The tags within that property take the form of key-value pairs, with each tag value appearing as an array containing a single string:

{
    "5be1f2ecc73c1714ceba66e2": {
        "imsOrg": "{ORG_ID}",
        "tags": {
            "sampleTag": [
                "123456"
            ],
            "secondTag": [
                "sample_tag_value"
            ]
        },
        "name": "Sample Dataset",
        "description": "Same dataset containing sample data.",
        "createdUser": "{CREATED_USER}",
        "createdClient": "{CREATED_CLIENT}",
        "updatedUser": "{UPDATED_USER}",
        "version": "1.0.0",
        "created": 1541534444286,
        "updated": 1541534444286
    }
}

API format

Values for the tags parameter take the form of key-value pairs, using the format {TAG_NAME}:{TAG_VALUE}. Multiple key-value pairs can be provided in the form of a comma-separated list. When multiple tags are provided, an AND relationship is assumed.

The parameter supports wildcard characters (*) for tag values. For example, a search string of test* returns any object where the tag value begins with “test”. A search string consisting solely of a wildcard can be used to filter objects based on whether or not they contain a specific tag, regardless of its value.

GET /{OBJECT_TYPE}?tags={TAG_NAME}:{TAG_VALUE}
GET /{OBJECT_TYPE}?tags={TAG_NAME_1}:{TAG_VALUE_1},{TAG_NAME_2}:{TAG_VALUE_2}
GET /{OBJECT_TYPE}?tags={TAG_NAME}:{TAG_VALUE}*
GET /{OBJECT_TYPE}?tags={TAG_NAME}:*

Request

The following request retrieves a list of datasets, filtering by one tag having a specific value AND the second tag being present.

curl -X GET \
  'https://platform.adobe.io/data/foundation/catalog/dataSets?tags=sampleTag:123456,secondTag:* \
  -H 'Authorization: Bearer {ACCESS_TOKEN}' \
  -H 'x-api-key: {API_KEY}' \
  -H 'x-gw-ims-org-id: {ORG_ID}' \
  -H 'x-sandbox-name: {SANDBOX_NAME}'

Response

A successful response returns a list of datasets that contain sampleTag with a value of “123456”, AND secondTag with any value. Unless a limit is also specified, the response contains a maximum of 20 objects.

{
    "5b67f4dd9f6e710000ea9da4": {
            "version": "1.0.2",
            "imsOrg": "{ORG_ID}",
            "name": "Example Dataset 1",
            "created": 1533539550237,
            "updated": 1533539552416,
            "createdClient": "{API_KEY}",
            "createdUser": "{USER_ID}",
            "updatedUser": "{USER_ID}",
            "tags": {
                "sampleTag": [
                    "123456"
                ],
                "secondTag": [
                    "Example tag value"
                ]
            },
    },
    "5b1e3c867e6d2600003d5b49": {
            "version": "1.0.0",
            "imsOrg": "{ORG_ID}",
            "name": "Example Dataset 2",
            "created": 1533539550237,
            "updated": 1533539552416,
            "createdClient": "{API_KEY}",
            "createdUser": "{USER_ID}",
            "updatedUser": "{USER_ID}",
            "tags": {
                "sampleTag": [
                    "123456"
                ],
                "secondTag": [
                    "A different tag value"
                ],
                "anotherTag": [
                    "2.0"
                ]
            },
    }
}

Filter by date range

Some endpoints in the Catalog API have query parameters that allow for ranged queries, most often in the case of dates.

API format

GET /batches?createdAfter={TIMESTAMP_1}&createdBefore={TIMESTAMP_2}

Request

The following request retrieves a list of batches created during the month of April 2019.

curl -X GET \
  'https://platform.adobe.io/data/foundation/catalog/batches?createdAfter=1554076800000&createdBefore=1556668799000' \
  -H 'Authorization: Bearer {ACCESS_TOKEN}' \
  -H 'x-api-key: {API_KEY}' \
  -H 'x-gw-ims-org-id: {ORG_ID}' \
  -H 'x-sandbox-name: {SANDBOX_NAME}'

Response

A successful response contains a list of Catalog objects that fall within the specified date range. Unless a limit is also specified, the response contains a maximum of 20 objects.

{
    "5b67f4dd9f6e710000ea9da4": {
            "version": "1.0.2",
            "imsOrg": "{ORG_ID}",
            "name": "Example Dataset 1",
            "created": 1554930967705,
            "updated": 1554931119718,
            "createdClient": "{API_KEY}",
            "createdUser": "{USER_ID}",
            "updatedUser": "{USER_ID}",
    },
    "5b1e3c867e6d2600003d5b49": {
            "version": "1.0.0",
            "imsOrg": "{ORG_ID}",
            "name": "Example Dataset 2",
            "created": 1554974386247,
            "updated": 1554974386268,
            "createdClient": "{API_KEY}",
            "createdUser": "{USER_ID}",
            "updatedUser": "{USER_ID}",
    }
}

Sort by property

The orderBy query parameter allows you to sort (order) response data based on a specified property value. This parameter requires a “direction” (asc for ascending or desc for descending), followed by a colon (:) and then a property to sort the results by. If a direction is not specified, the default direction is ascending.

Multiple sorting properties can be provided in a comma-separated list. If the first sorting property produces multiple objects that contain the same value for that property, the second sorting property is then used to further sort those matching objects.

For example, consider the following query: orderBy=name,desc:created. Results are sorted in ascending order based on the first sorting property, name. In cases where multiple records share the same name property, those matching records are then sorted by the second sorting property, created. If no returned records share the same name, the created property does not factor into the sorting.

API format

GET /{OBJECT_TYPE}?orderBy=asc:{PROPERTY_NAME}
GET /{OBJECT_TYPE}?orderBy=desc:{PROPERTY_NAME}
GET /{OBJECT_TYPE}?orderBy={PROPERTY_NAME_1},desc:{PROPERTY_NAME_2}

Request

The following request retrieves a list of datasets sorted by their name property. If any datasets share the same name, those datasets will in turn be ordered by their updated property in descending order.

curl -X GET \
  'https://platform.adobe.io/data/foundation/catalog/dataSets?orderBy=name,desc:updated' \
  -H 'Authorization: Bearer {ACCESS_TOKEN}' \
  -H 'x-api-key: {API_KEY}' \
  -H 'x-gw-ims-org-id: {ORG_ID}' \
  -H 'x-sandbox-name: {SANDBOX_NAME}'

Response

A successful response contains a list of Catalog objects that are sorted according to the orderBy parameter. Unless a limit is also specified, the response contains a maximum of 20 objects.

{
    "5b67f4dd9f6e710000ea9da4": {
            "version": "1.0.2",
            "imsOrg": "{ORG_ID}",
            "name": "0405",
            "created": 1554930967705,
            "updated": 1554931119718,
            "createdClient": "{API_KEY}",
            "createdUser": "{USER_ID}",
            "updatedUser": "{USER_ID}",
    },
    "5b1e3c867e6d2600003d5b49": {
            "version": "1.0.3",
            "imsOrg": "{ORG_ID}",
            "name": "AAM Dataset",
            "created": 1554974386247,
            "updated": 1554974386268,
            "createdClient": "{API_KEY}",
            "createdUser": "{USER_ID}",
            "updatedUser": "{USER_ID}",
    },
    "5cd3a129ec106214b722a939": {
            "version": "1.0.2",
            "imsOrg": "{ORG_ID}",
            "name": "AAM Dataset",
            "created": 1554028394852,
            "updated": 1554130582960,
            "createdClient": "{API_KEY}",
            "createdUser": "{USER_ID}",
            "updatedUser": "{USER_ID}",
    }
}

Filter by property

Catalog provides two methods of filtering by property, which are further outlined in the sections that follow:

Using simple filters using-simple-filters

Simple filters allow you to filter responses based on specific property values. A simple filter takes the form of {PROPERTY_NAME}={VALUE}.

For example, the query name=exampleName returns only objects whose name property contains a value of “exampleName”. By contrast, the query name=!exampleName returns only objects whose name property is not “exampleName”.

In addition, simple filters support the ability to query for multiple values for a single property. When multiple values are provided, the response returns objects whose property matches any of the values in the provided list. You can invert a multi-value query by prefixing a ! character to the list, returning only objects whose property value is not in the provided list (for example, name=!exampleName,anotherName).

API format

GET /{OBJECT_TYPE}?{PROPERTY_NAME}={VALUE}
GET /{OBJECT_TYPE}?{PROPERTY_NAME}=!{VALUE}
GET /{OBJECT_TYPE}?{PROPERTY_NAME}={VALUE_1},{VALUE_2},{VALUE_3}
GET /{OBJECT_TYPE}?{PROPERTY_NAME}=!{VALUE_1},{VALUE_2},{VALUE_3}

Request

The following request retrieves a list of datasets, filtered to include only datasets whose name property has a value of “exampleName” or “anotherName”.

curl -X GET \
  'https://platform.adobe.io/data/foundation/catalog/dataSets?name=exampleName,anotherName' \
  -H 'Authorization: Bearer {ACCESS_TOKEN}' \
  -H 'x-api-key: {API_KEY}' \
  -H 'x-gw-ims-org-id: {ORG_ID}' \
  -H 'x-sandbox-name: {SANDBOX_NAME}'

Response

A successful response contains a list of datasets, excluding any datasets whose name is “exampleName” or “anotherName”. Unless a limit is also specified, the response contains a maximum of 20 objects.

{
    "5b67f4dd9f6e710000ea9da4": {
            "version": "1.0.2",
            "imsOrg": "{ORG_ID}",
            "name": "exampleName",
            "created": 1554930967705,
            "updated": 1554931119718,
            "createdClient": "{API_KEY}",
            "createdUser": "{USER_ID}",
            "updatedUser": "{USER_ID}",
    },
    "5b1e3c867e6d2600003d5b49": {
            "version": "1.0.3",
            "imsOrg": "{ORG_ID}",
            "name": "anotherName",
            "created": 1554974386247,
            "updated": 1554974386268,
            "createdClient": "{API_KEY}",
            "createdUser": "{USER_ID}",
            "updatedUser": "{USER_ID}",
    }
}

Using the property parameter using-the-property-parameter

The property query parameter provides more flexibility for property-based filtering than simple filters. In addition to filtering based on whether a property has a specific value, the property parameter can use other comparison operators (such as “more-than” (>) and “less-than” (<)) as well as regular expressions to filter by property values. It can also filter by whether or not a property exists, regardless of its value.

Use an ampersand (&) to combine multiple filters and refine your query in a single request. When you filter by multiple fields, an AND relationship is applied by default.

GET /datasets?property=id==abc123&property=name!=test

If you use multiple property parameters on the same field, only the last specified parameter takes effect.

The property parameter can accept any level object properties. sampleKey can be used for filtering using ?properties=subItem.sampleKey.

{
  "5ba9452f7de80400007fc52a": {
      "name": "Sample Dataset",
      "description": "Sample dataset containing important data",
      "subitem": {
          "sampleKey": "sampleValue"
      }
  }
}

API format

GET /{OBJECT_TYPE}?property={CONDITION}

The value of the property parameter supports several different kinds of conditional expressions. The following table outlines the basic syntax for supported expressions:

Request

The following request will return any datasets with a version number greater than 1.0.3.

curl -X GET \
  https://platform.adobe.io/data/foundation/catalog/dataSets?property=version>1.0.3 \
  -H 'Authorization: Bearer {ACCESS_TOKEN}' \
  -H 'x-api-key: {API_KEY}' \
  -H 'x-gw-ims-org-id: {ORG_ID}' \
  -H 'x-sandbox-name: {SANDBOX_NAME}'

Response

A successful response contains a list of datasets whose version numbers are greater than 1.0.3. Unless a limit is also specified, the response contains a maximum of 20 objects.

{
    "5b67f4dd9f6e710000ea9da4": {
            "version": "1.1.2",
            "imsOrg": "{ORG_ID}",
            "name": "sampleDataset",
            "created": 1554930967705,
            "updated": 1554931119718,
            "createdClient": "{API_KEY}",
            "createdUser": "{USER_ID}",
            "updatedUser": "{USER_ID}",
    },
    "5b1e3c867e6d2600003d5b49": {
            "version": "1.0.6",
            "imsOrg": "{ORG_ID}",
            "name": "exampleDataset",
            "created": 1554974386247,
            "updated": 1554974386268,
            "createdClient": "{API_KEY}",
            "createdUser": "{USER_ID}",
            "updatedUser": "{USER_ID}",
    },
    "5cd3a129ec106214b722a939": {
            "version": "1.0.4",
            "imsOrg": "{ORG_ID}",
            "name": "anotherDataset",
            "created": 1554028394852,
            "updated": 1554130582960,
            "createdClient": "{API_KEY}",
            "createdUser": "{USER_ID}",
            "updatedUser": "{USER_ID}",
    }
}

Filter arrays with the property parameter filter-arrays

Use equality and inequality operators to include or exclude specific values when filtering results based on array properties.

Equality filters equality-filters

To filter an array field by multiple values, use separate property parameters for each value. Use equality filters to return only the entries in the array data that match the specified values.

The example query below only returns results from the array that includes both val1 and val2.

API format

GET /{OBJECT_TYPE}?property=arrayField=val1&property=arrayField=val2

Inequality filters inequality-filters

Use inequality operators (!=) on an array field to exclude any entries in the data where the array contains the specified values.

API format

GET /{OBJECT_TYPE}?property=arrayField!=val1&property=arrayField!=val2

This query returns documents where arrayField does not contain val1 or val2.

Equality and inequality filter limitations equality-inequality-limitations

You cannot apply both equality (=) and inequality (!=) to the same field in a single query.