SegmentNotFoundException in AEM logs and instance aren’t working

Download the oak-run jar file from org/apache/jackrabbit/oak-run.

Stop AEM.

Run this command:
java -jar oak-run-*.jar check --bin=-1 crx-quickstart/repository/segmentstore/

This command searches backwards through the revisions until it finds a consistent one:

14:00:30.783 [ main] INFO o.a.j.o.p.s.f.t.ConsistencyChecker - Found latest good revision xxxx000x-ba53-4a1b-aa1b-0xx000x000xx:234880

Note: In case the ConsistencyChecker fails, go to the next section.

Revert the repository to this revision by editing the command given below.
/crx-quickstart/repository/segmentstore/journal.log.

Delete all lines after the line containing the latest good revision. To find out what date and time need to be reverted to the repository, run this command in the segmentstore folder (replace xxxx000x-ba53-4a1b-aa1b-0xx000x000xx with the latest good revision in the journal.log):

find . -type f -name "data*.tar" -exec sh -c "tar -tvf {} |grep xxxx000x-ba53-4a1b-aa1b-0xx000x000xx" \; -print

The output would show an approximate date and time of that revision.

Remove all ./crx-quickstart/repository/segmentstore/*.bak files.

If AEM 6.0 is being used, then download the oak-run version matching to the installed version in AEM for the remaining steps. Download it from this link: org/apache/jackrabbit/oak-run.

Run the checkpoint clean-up to remove orphaned checkpoints:
java -jar oak-run-*.jar checkpoints ./crx-quickstart/repository/segmentstore rm-unreferenced.

Finally, compact the repository:
java -jar oak-run-*.jar compact ./crx-quickstart/repository/segmentstore/.

Stop AEM.

Use the Oak run console and load the childCount groovy script to identify the corrupt nodes in the segment store:

• Load the oak-run console shell:

  java -jar oak-run-*.jar console crx-quickstart/repository/segmentstore

• Run the two commands below in the shell to load the script and run it:

  : load https://gist.githubusercontent.com/stillalex/e7067bcb86c89bef66c8/raw/d7a5a9b839c3bb0ae5840252022f871fd38374d3/childCount.groovy

  countNodes(session.workingNode)

• This results in the following output indicating the path to the corrupted node(s):

  code language-none
  21:21:42.029 [ main] ERROR o.a.j.o.p.segment.SegmentTracker - Segment not found: 63ae05a4-b506-445c-baa2-cfa1b13b6e2f. Creation date delta is 3 ms. warning unable to read node /content/dam/test.txt/jcr:content/renditions/original/jcr:content

  • In some cases the issue is linked to binary properties and the childCount groovy script is unable to locate any corrupted nodes. In these cases you can use the following command instead which will read the first 1024 bytes for every binary encountered during the traversal.

  (Please note this command will be slower and should only be used when the solution above doesn’t return the expected results):

  countNodes(session.workingNode,true)

Remove all the identified corrupted nodes listed in the output of the last command using rmNodes.groovy.

• Load the oak-run console shell:

  java -jar oak-run-*.jar console crx-quickstart/repository/segmentstore

• Load the groovy script:

  :load https://gist.githubusercontent.com/stillalex/43c49af065e3dd1fd5bf/raw/9e726a59f75b46e7b474f7ac763b0888d5a3f0c3/rmNode.groovy

• Run the rmNode command to remove the corrupt node, replace /path/to/corrupt/node with the path to the corrupt node you need to remove.

  rmNode(session, "/path/to/corrupt/node")

  Where the corrupt node path is the path obtained in step 2, for example:

  "/content/dam/test.txt/jcr:content/renditions/original/jcr:content/"

Note: When using oak-run.jar version 1.6.13 and up, set --read-write JVM parameter if you run into an error such as:

rmNode(session,"/path/to/corrupt/node") Removing node /path/to/corrupt/node ERROR java.lang.UnsupportedOperationException:
Cannot write to read-only store at org.apache.jackrabbit.oak.segment.SegmentWriterBuilder$1.execute
(SegmentWriterBuilder.java:171) at org.apache.jackrabbit.oak.segment.SegmentWriter.writeNode (SegmentWriter.java:318) at
org.apache.jackrabbit.oak.segment.SegmentNodeBuilder.getNodeState (SegmentNodeBuilder.java:111) at
org.apache.jackrabbit.oak.segment.SegmentNodeStore$Commit. (SegmentNodeStore.java:581) at
org.apache.jackrabbit.oak.segment.SegmentNodeStore.merge (SegmentNodeStore.java:333) at
org.apache.jackrabbit.oak.spi.state.NodeStore$merge.call (Unknown Source) at groovysh_evaluate.rmNode (groovysh_evaluate:11)

Repeat step 3 for all nodes found in step 2. This above rmNode command should return true for the corrupt path, which means it deleted it. Make sure these found three corrupt paths get deleted by re-running the rmNode command on those paths. Next run it should return false. If the same paths are there in repository, then it is advisable to use the patched version of oak-run jar i.e. oak-run-1.2.18-NPR-00000.

What does patched version of Oak run Jar do?

This version of jar skips unreadable binaries on compaction replacing them with 0-byte binaries and logging the exception and the path to syserr (Error Messages window). The compacted repository should then pass oak-run check, the node count script and also be able to compact again using a non-patched oak-run.

Perform a checkpoint cleanup by listing checkpoints using below. If there are more than one checkpoint, clean it up:

nohup java -Xmx4096m -jar oak-run-1.2.18.jar checkpoints /app/AEM6/author/crx-quickstart/repository/segmentstore rm-all >> nohup.out &

Run an offline compaction. Learn how to run offline compaction here.

Start the server and wait for the indexing completion.