Workaround and fix index corruption with AEM/AEM Forms

Description




Experiencing an index corruption should be a very unusual case as a lot of work has been put into avoiding it, but should it happen you would see log entries similar to:

17.01.2019 17:55:47.374 *WARN* pool-118-thread-4 org.apache.jackrabbit.oak.plugins.index.AsyncIndexUpdate async The index update failed``org.apache.jackrabbit.oak.api.CommitFailedException: OakMerge0001: OakMerge0001: Failed to merge changes to the underlying store (retries 5, 5169 ms)``at org.apache.jackrabbit.oak.spi.state.AbstractNodeStoreBranch.merge0(AbstractNodeStoreBranch.java:345)``at org.apache.jackrabbit.oak.plugins.document.DocumentNodeStoreBranch.merge(DocumentNodeStoreBranch.java:156)``at org.apache.jackrabbit.oak.plugins.document.DocumentRootBuilder.merge(DocumentRootBuilder.java:160)``at org.apache.jackrabbit.oak.plugins.document.DocumentNodeStore.merge(DocumentNodeStore.java:1588)``at org.apache.jackrabbit.oak.plugins.index.AsyncIndexUpdate.mergeWithConcurrencyCheck(AsyncIndexUpdate.java:581)``at org.apache.jackrabbit.oak.plugins.index.AsyncIndexUpdate.updateIndex(AsyncIndexUpdate.java:525)``at org.apache.jackrabbit.oak.plugins.index.AsyncIndexUpdate.runWhenPermitted(AsyncIndexUpdate.java:431)``at org.apache.jackrabbit.oak.plugins.index.AsyncIndexUpdate.run(AsyncIndexUpdate.java:323)``at org.apache.sling.commons.scheduler.impl.QuartzJobExecutor.execute(QuartzJobExecutor.java:115)``at org.quartz.core.JobRunShell.run(JobRunShell.java:202)``at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1164)``at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:634)

Resolution

If you face this situation, probably the fastest way to solve this issue is by deleting the index data that is contained in the repository and reindex the corrupted index

NOTE: Reindexing is a time and resource consuming action

for this examples we will use the index "workflowDataLucene"

Delete the index data: 

The index data is located in the repository belows /oak:index/workflowDataLucene as hidden nodes, therefore in order to delete the corrupted index, you need to:

  1. Copy the index deffinition (only) to a new location
  2. Delete the whole /oak:index/workflowDataLucene node structure
  3. Recreate /oak:index/workflowDataLucene and tirgger reindex

Workaround:

As a workaround, instead of deleteing the index, you can deactivate it and create a new index with the same definition:

  1. Create a new index with the same index definition as the corrupted one
  2. Trigger reindex on the new index
  3. Deactivate the corrupted index by changing the type property to “disabled”

once this changes have been done every query will go to the new index

Delete the data from RDB:

As a last resort and if you run on RDBmk, you could analyze to delete the index data directly from the database with as query similar to:

DELETE FROM NODES``WHERE
ID =``'2:/oak:index/workflowDataLucene' OR``ID``LIKE '3:/oak:index/workflowDataLucene/%' OR``ID``LIKE '4:/oak:index/workflowDataLucene/%' OR``ID``LIKE '5:/oak:index/workflowDataLucene/%' OR``ID``LIKE '6:/oak:index/workflowDataLucene/%' OR``ID``LIKE '7:/oak:index/workflowDataLucene/%' OR``ID``LIKE '8:/oak:index/workflowDataLucene/%' OR``ID``LIKE '9:/oak:index/workflowDataLucene/%' OR``ID``LIKE '10:/oak:index/workflowDataLucene/%' OR``  
ID``LIKE '5:p/oak:index/workflowDataLucene/%' OR``ID``LIKE '6:p/oak:index/workflowDataLucene/%' OR``ID``LIKE '7:p/oak:index/workflowDataLucene/%' OR``ID``LIKE '8:p/oak:index/workflowDataLucene/%' OR``ID``LIKE '9:p/oak:index/workflowDataLucene/%' OR``ID``LIKE '10:p/oak:index/workflowDataLucene/%' OR``ID``LIKE '11:p/oak:index/workflowDataLucene/%' OR``ID``LIKE '12:p/oak:index/workflowDataLucene/%' OR
 

If you decide to go this path, make sure to verify the query before running it as it might require some changes

NOTE: AEM must be shutdown and local persistent cache and index files (crx-quickstart/repository/cache and crx-quickstart/repository/index) deleted before starting AEM after this deletion.

On this page