Fix inconsistencies in repository when SegmentNotFound reported in AEM 6.x

Last update: November 21, 2023

If you have inconsistencies in the repository (repo) when SegmentNotFound reported, run a consistency check on the repo, find the last good revision (healthy state), and revert back to it.

Description

Environment

Adobe Experience Manager (AEM) 6.x

Issue/Symptoms

The exceptions for SegmentNotFound are observed in the logs.

Example:

*ERROR* [ FelixStartLevel] org.apache.sling.event [ org.apache.sling.event.impl.jobs.queues.QueueManager(1431)] The activate method has thrown an exception (org.apache.jackrabbit.oak.plugins.segment.SegmentNotFoundException: Segment da5bcb95-d00a-4c04-a9d9-0f10f2b14e5e not found)
**ERROR* [ pool-6-thread-3] org.apache.sling.commons.scheduler.impl.QuartzScheduler Exception during job execution of org.apache.jackrabbit.oak.plugins.index.AsyncIndexUpdate@1dc173f9 : Segment e669f30b-e886-4b7a-b161-56432601ec6b not found

org.apache.jackrabbit.oak.plugins.segment.SegmentNotFoundException: Segment e669f30b-e886-4b7a-b161-56432601ec6b not found*

Resolution

Run a consistency check on the repo and find the last good revision (healthy state) and revert back to it.

Follow these steps:

Download a version of oak-run that matches your oak core version from https://mvnrepository.com/artifact/org.apache.jackrabbit/oak-run
To revert a corrupt segment store to its latest good state, change into CQ’s working directory (the one containing the crx-quickstartfolder) and back up all files in ./crx-quickstart/repository/segmentstore/.
Run the consistency check,

java -Xmx6000m -jar oak-run-*.jar check --bin=-1 /path/to/crx-quickstart/repository/segmentstore

This searches backward through the revisions until it finds a consistent one:

Look for a message like the one below:

[ main] INFO o.a.j.o.p.s.f.t.ConsistencyChecker - Found latest good revision afdb922d-ba53-4a1b-aa1b-1cb044b535cf:234880
Revert the repo to this revision by editing ./crx-quickstart/repository/segmentstore/journal.log and deleting all lines after the line containing the latest good revision.
Remove all ./crx-quickstart/repository/segmentstore/*.bak files.
Run checkpoint clean-up to remove orphaned checkpoints:

java -Xmx6000m -jar oak-run-*.jar checkpoints /path/to/crx-quickstart/repository/segmentstore rm-unreferenced
Finally, compact the repo:

java -Xmx6000m -jar oak-run-*.jar compact /path/to/crx-quickstart/repository/segmentstore/

Cause
Due to some older issues in Oak or some inconsistencies in the repository, a segment can go missing, and the repository might be inconsistent.

recommendation-more-help