Indexing best practices in AEM

Understand the search requirements and check if the OOTB indexes can support the search requirements. Use Query Performance Tool, available at local SDK and AEMCS via the Developer Console or https://author-pXXXX-eYYYY.adobeaemcloud.com/ui#/aem/libs/granite/operations/content/diagnosistools/queryPerformance.html?appId=aemshell.

Define an optimal query, use the optimizing queries flow chart and JCR Query Cheat Sheet for reference.

If the OOTB indexes cannot support the search requirements, you have two options. However, review the Tips for Creating Efficient Indexes

• Customize the OOTB index: Preferred option as it is easy to maintain and upgrade.
• Fully custom index: Only if the above option does not work.

In AEMCS, when customizing the OOTB index use <OOTBIndexName>-<productVersion>-custom-<customVersion> naming convention. For example, cqPageLucene-custom-1 or damAssetLucene-8-custom-1. This helps to merge the customized index definition whenever the OOTB index is updated. See Changes to Out-of-the-Box Indexes for more details.

In AEM 6.X, the above naming does not work, however simply update the OOTB index with necessary properties in the indexRules node.

Always copy the latest OOTB index definition from the AEM instance using the CRX DE Package Manager (https://experienceleague.adobe.com/crx/packmgr/?lang=en), rename it and add customizations inside the XML file.

Store index definition into the AEM project at ui.apps/src/main/content/jcr_root/_oak_index and deploy it using Cloud Manager CI/CD pipelines. See Deploying Custom Index Definitions for more details.

When creating a fully custom index, use <prefix>.<customIndexName>-<version>-custom-<customVersion> naming convention. For example, wknd.adventures-1-custom-1. This helps to avoid naming conflicts. Here, wknd is the prefix and adventures is the custom index name. This convention is applicable for both AEM 6.X and AEMCS and helps to prepare for future migration to AEMCS.

AEMCS only supports Lucene indexes, so to prepare for future migration to AEMCS, always use Lucene indexes. See Lucene Indexes vs Property Indexes for more details.

Avoid creating a custom index on the same node type as the OOTB index. Instead, customize the OOTB index with necessary properties in the indexRules node. For example, do not create a custom index on the dam:Asset node type but customize the OOTB damAssetLucene index. It has been a common root cause of performance and functional issues.

Also, avoid adding multiple node types for example cq:Page and cq:Tag under the indexing rules (indexRules) node. Instead, create separate indexes for each node type.

As mentioned in above section, store index definition into the AEM project at ui.apps/src/main/content/jcr_root/_oak_index and deploy it using Cloud Manager CI/CD pipelines. See Deploying Custom Index Definitions for more details.

The index definition guidelines are:

• The node type (jcr:primaryType) should be oak:QueryIndexDefinition
• The index type (type) should be lucene
• The async property (async) should be async,nrt
• Use includedPaths and avoid excludedPaths property. Always set queryPaths value to the same value as includedPaths value.
• To enforce the path restriction, use evaluatePathRestrictions property and set it to true.
• Use tags property to tag the index and while querying specify this tags value to use the index. The general query syntax is <query> option(index tag <tagName>).

/oak:index/wknd.adventures-1-custom-1
    - jcr:primaryType = "oak:QueryIndexDefinition"
    - type = "lucene"
    - compatVersion = 2
    - async = ["async", "nrt"]
    - includedPaths = ["/content/wknd"]
    - queryPaths = ["/content/wknd"]
    - evaluatePathRestrictions = true
    - tags = ["customAdvSearch"]
...

Index Manager available at http://host:port/libs/granite/operations/content/diagnosistools/indexManager.html for getting index info like type, last updated, size.

Detailed logging of Oak query and indexing-related Java™ packages like org.apache.jackrabbit.oak.plugins.index, org.apache.jackrabbit.oak.query, and com.day.cq.search via http://host:port/system/console/slinglog for troubleshooting.

JMX MBean of IndexStats type available at http://host:port/system/console/jmx for getting index info like status, progress, or statistics related to asynchronous indexing. It also provides FailingIndexStats, if there are no results here, means that no indexes are corrupt. AsyncIndexerService marks any index that fails to update for 30 minutes (configurable) as corrupt and stops indexing them. If a query is not giving expected results, it is helpful for developers to check this before proceeding with reindexing as reindexing is computationally expensive and time consuming.

JMX MBean of LuceneIndex type available at http://host:port/system/console/jmx for Lucene Index statistics like size, number of documents per index definition.

JMX MBean of QueryStat type available at http://host:port/system/console/jmx for Oak Query Statistics including slow and popular queries with details like query, execution time.