Workaround and fix index corruption with AEM/AEM Forms

If someone experiences the issue of index corruption in the AEM/AEM forms, this article will help you resolve the issue by showing how to delete the index data that is contained in the repository and reindex the corrupted index.

Description

Environment

Experience Manager Forms

Issue/Symptoms

Experiencing an index corruption is a very unusual case. But if it happens, one will see log entries similar to:
17.01.2019 17:55:47.374 *WARN* [ pool-118-thread-4] org.apache.jackrabbit.oak.plugins.index.AsyncIndexUpdate [ async] The index update failed``org.apache.jackrabbit.oak.api.CommitFailedException: OakMerge0001: OakMerge0001: Failed to merge changes to the underlying store (retries 5, 5169 ms)``at org.apache.jackrabbit.oak.spi.state.AbstractNodeStoreBranch.merge0(AbstractNodeStoreBranch.java:345)``at org.apache.jackrabbit.oak.plugins.document.DocumentNodeStoreBranch.merge(DocumentNodeStoreBranch.java:156)``at org.apache.jackrabbit.oak.plugins.document.DocumentRootBuilder.merge(DocumentRootBuilder.java:160)``at org.apache.jackrabbit.oak.plugins.document.DocumentNodeStore.merge(DocumentNodeStore.java:1588)``at org.apache.jackrabbit.oak.plugins.index.AsyncIndexUpdate.mergeWithConcurrencyCheck(AsyncIndexUpdate.java:581)``at org.apache.jackrabbit.oak.plugins.index.AsyncIndexUpdate.updateIndex(AsyncIndexUpdate.java:525)``at org.apache.jackrabbit.oak.plugins.index.AsyncIndexUpdate.runWhenPermitted(AsyncIndexUpdate.java:431)``at org.apache.jackrabbit.oak.plugins.index.AsyncIndexUpdate.run(AsyncIndexUpdate.java:323)``at org.apache.sling.commons.scheduler.impl.QuartzJobExecutor.execute(QuartzJobExecutor.java:115)``at org.quartz.core.JobRunShell.run(JobRunShell.java:202)``at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1164)``at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:634)

Resolution

If someone face this situation, probably the fastest way to solve this issue is by deleting the index data that is contained in the repository and reindex the corrupted index.

NOTE: Reindexing is a time and resource consuming action for these examples we will use the index "workflowDataLucene"

Delete the index data: 

The index data is located in the repository belows /oak:index/workflowDataLucene as hidden nodes, therefore in order to delete the corrupted index, you need to:

1. Copy the index definition (only) to a new location.
2. Delete the whole /oak:index/workflowDataLucene node structure.
3. Recreate /oak:index/workflowDataLucene and tirgger reindex.

Workaround:

As a workaround, instead of deleteing the index, you can deactivate it and create a new index with the same definition:

1. Create a new index with the same index definition as the corrupted one
2. Trigger reindex on the new index
3. Deactivate the corrupted index by changing the type property to “disabled”

once this changes have been done every query will go to the new index

Delete the data from RDB:

As a last resort and if you run on RDBmk, you could analyze to delete the index data directly from the database with as query similar to:

DELETE FROM NODES``WHERE
ID =``'2:/oak:index/workflowDataLucene' OR``ID``LIKE '3:/oak:index/workflowDataLucene/%' OR``ID``LIKE '4:/oak:index/workflowDataLucene/%' OR``ID``LIKE '5:/oak:index/workflowDataLucene/%' OR``ID``LIKE '6:/oak:index/workflowDataLucene/%' OR``ID``LIKE '7:/oak:index/workflowDataLucene/%' OR``ID``LIKE '8:/oak:index/workflowDataLucene/%' OR``ID``LIKE '9:/oak:index/workflowDataLucene/%' OR``ID``LIKE '10:/oak:index/workflowDataLucene/%' OR``  
ID``LIKE '5:p/oak:index/workflowDataLucene/%' OR``ID``LIKE '6:p/oak:index/workflowDataLucene/%' OR``ID``LIKE '7:p/oak:index/workflowDataLucene/%' OR``ID``LIKE '8:p/oak:index/workflowDataLucene/%' OR``ID``LIKE '9:p/oak:index/workflowDataLucene/%' OR``ID``LIKE '10:p/oak:index/workflowDataLucene/%' OR``ID``LIKE '11:p/oak:index/workflowDataLucene/%' OR``ID``LIKE '12:p/oak:index/workflowDataLucene/%' OR
 

If you decide to go this path, make sure to verify the query before running it as it might require some changes

NOTE: AEM must be shutdown and local persistent cache and index files (< crx-quickstart> /repository/cache and < crx-quickstart> /repository/index) deleted before starting AEM after this deletion.