AEM as a Cloud Service pipeline stuck in RUNNING state due to custom Oak/Lucene index issues
This article explains how to resolve situations where an Adobe Experience Manager (AEM) as a Cloud Service deployment pipeline remains stuck in the RUNNING state due to issues with custom Oak/Lucene indexes.
Description description
Environment
Adobe Experience Manager as a Cloud Service (AEMaaCS), all versions using Cloud Manager pipelines.
Issue/Symptoms
- The Cloud Manager deployment pipeline remains in the RUNNING state for an extended period and does not complete.
- The environment readiness check is blocked, preventing the pipeline from progressing.
- Indexing job logs show errors such as:
Error while processing an index write operation java.io.IOException: Invalid byte range request for [ /oak:index/cqPageLucene-custom-1] ...
- Filesystem errors such as:
java.nio.file.NoSuchFileException
- Custom index nodes, such as
/oak:index/cqPageLucene-custom-1and/oak:index/damAssetLucene-11-custom-4, appear in the repository even if they are not present in the current codebase. - Deleting these index nodes manually in CRXDE is only temporary because they can reappear after redeployment.
Cause
The pipeline is blocked because the environment readiness check cannot complete due to a corrupted or inconsistent custom Oak/Lucene index. This prevents the indexing job from finishing, leaving the pipeline stuck in the RUNNING state. Manual deletion of index nodes is not persistent; proper removal requires updating the codebase and using the supported index management approach.
Resolution resolution
Follow the steps below to resolve the issue:
- Review the indexing job logs and identify errors related to custom indexes, such as
/oak:index/cqPageLucene-custom-1. Confirm whether the index nodes exist in the codebase underui.apps/src/main/content/jcr_root/_oak_index/or in adiff.jsonfile. - Search the repository for definitions of the problematic indexes, such as
cqPageLucene-custom-1ordamAssetLucene-11-custom-4. Delete these definitions from_oak_indexor any index content packages, and then commit the changes. - Express all custom index changes in
/oak:index/diff.index/diff.jsonby using the Simplified Index Management model. Deploy the updated code through Cloud Manager. - After cleaning up the codebase and confirming that only the intended customizations exist in
diff.json, redeploy through Cloud Manager. The pipeline should complete successfully if no index corruption remains. - If the issue persists, review the logs for filesystem errors or Lucene segment corruption. Rebuild the affected index by removing the problematic runtime node and redeploying.
- After redeployment, the pipeline should complete successfully and the environment should be marked as ready.
Notes
- The platform generates new
*-custom-*nodes as needed and automatically retires unused older versions over time. - Avoid deleting index nodes manually in CRXDE as a permanent solution because the nodes can be recreated during later deployments if the codebase or
diff.jsonstill references them.
experience-cloud-kcs-help-kbarticles