AEM as a Cloud Service pipeline stuck in RUNNING state due to custom Oak/Lucene index issues

This article explains how to resolve situations where an Adobe Experience Manager (AEM) as a Cloud Service deployment pipeline remains stuck in the RUNNING state due to issues with custom Oak/Lucene indexes.

Description description

Environment

Adobe Experience Manager as a Cloud Service (AEMaaCS), all versions using Cloud Manager pipelines.

Issue/Symptoms

  • The Cloud Manager deployment pipeline remains in the RUNNING state for an extended period and does not complete.
  • The environment readiness check is blocked, preventing the pipeline from progressing.
  • Indexing job logs show errors such as:
Error while processing an index write operation java.io.IOException: Invalid byte range request for [ /oak:index/cqPageLucene-custom-1] ...
  • Filesystem errors such as:
java.nio.file.NoSuchFileException
  • Custom index nodes, such as /oak:index/cqPageLucene-custom-1 and /oak:index/damAssetLucene-11-custom-4, appear in the repository even if they are not present in the current codebase.
  • Deleting these index nodes manually in CRXDE is only temporary because they can reappear after redeployment.

Cause

The pipeline is blocked because the environment readiness check cannot complete due to a corrupted or inconsistent custom Oak/Lucene index. This prevents the indexing job from finishing, leaving the pipeline stuck in the RUNNING state. Manual deletion of index nodes is not persistent; proper removal requires updating the codebase and using the supported index management approach.

Resolution resolution

Follow the steps below to resolve the issue:

  1. Review the indexing job logs and identify errors related to custom indexes, such as /oak:index/cqPageLucene-custom-1. Confirm whether the index nodes exist in the codebase under ui.apps/src/main/content/jcr_root/_oak_index/ or in a diff.json file.
  2. Search the repository for definitions of the problematic indexes, such as cqPageLucene-custom-1 or damAssetLucene-11-custom-4. Delete these definitions from _oak_index or any index content packages, and then commit the changes.
  3. Express all custom index changes in /oak:index/diff.index/diff.json by using the Simplified Index Management model. Deploy the updated code through Cloud Manager.
  4. After cleaning up the codebase and confirming that only the intended customizations exist in diff.json, redeploy through Cloud Manager. The pipeline should complete successfully if no index corruption remains.
  5. If the issue persists, review the logs for filesystem errors or Lucene segment corruption. Rebuild the affected index by removing the problematic runtime node and redeploying.
  6. After redeployment, the pipeline should complete successfully and the environment should be marked as ready.

Notes

  • The platform generates new *-custom-* nodes as needed and automatically retires unused older versions over time.
  • Avoid deleting index nodes manually in CRXDE as a permanent solution because the nodes can be recreated during later deployments if the codebase or diff.json still references them.

Content search and indexing

recommendation-more-help
experience-cloud-kcs-help-kbarticles