Fix inconsistencies in repository when SegmentNotFound reported in AEM 6.x

If you have inconsistencies in the repository (repo) when SegmentNotFound reported, run a consistency check on the repo, find the last good revision (healthy state), and revert back to it.

Description description

Environment

Adobe Experience Manager (AEM) 6.x

Issue/Symptoms

The exceptions for SegmentNotFound are observed in the logs.

Example:

  1. *ERROR* [ FelixStartLevel] org.apache.sling.event [ org.apache.sling.event.impl.jobs.queues.QueueManager(1431)] The activate method has thrown an exception (org.apache.jackrabbit.oak.plugins.segment.SegmentNotFoundException: Segment da5bcb95-d00a-4c04-a9d9-0f10f2b14e5e not found)

  2. **ERROR* [ pool-6-thread-3] org.apache.sling.commons.scheduler.impl.QuartzScheduler Exception during job execution of org.apache.jackrabbit.oak.plugins.index.AsyncIndexUpdate@1dc173f9 : Segment e669f30b-e886-4b7a-b161-56432601ec6b not found

    org.apache.jackrabbit.oak.plugins.segment.SegmentNotFoundException: Segment e669f30b-e886-4b7a-b161-56432601ec6b not found*

Resolution resolution

Run a consistency check on the repo and find the last good revision (healthy state) and revert back to it.

Follow these steps:

  1. Download a version of oak-run that matches your oak core version from https://mvnrepository.com/artifact/org.apache.jackrabbit/oak-run

  2. To revert a corrupt segment store to its latest good state, change into CQ’s working directory (the one containing the crx-quickstartfolder) and back up all files in ./crx-quickstart/repository/segmentstore/.

  3. Run the consistency check,

    java -Xmx6000m -jar oak-run-*.jar check --bin=-1 /path/to/crx-quickstart/repository/segmentstore

    This searches backward through the revisions until it finds a consistent one:

    Look for a message like the one below:

    [ main] INFO o.a.j.o.p.s.f.t.ConsistencyChecker - Found latest good revision afdb922d-ba53-4a1b-aa1b-1cb044b535cf:234880

  4. Revert the repo to this revision by editing ./crx-quickstart/repository/segmentstore/journal.log and deleting all lines after the line containing the latest good revision.

  5. Remove all ./crx-quickstart/repository/segmentstore/*.bak files.

  6. Run checkpoint clean-up to remove orphaned checkpoints:

    java -Xmx6000m -jar oak-run-*.jar checkpoints /path/to/crx-quickstart/repository/segmentstore rm-unreferenced

  7. Finally, compact the repo:

    java -Xmx6000m -jar oak-run-*.jar compact /path/to/crx-quickstart/repository/segmentstore/

Cause
Due to some older issues in Oak or some inconsistencies in the repository, a segment can go missing, and the repository might be inconsistent.

recommendation-more-help
3d58f420-19b5-47a0-a122-5c9dab55ec7f