Datastore inconsistency error obtaining “InputStream for blobId”

Install the latest CFP/SP.

You can find the latest hot fixes on Adobe Package Share.

Run datastore consistency check.

Using oak-run jar 1.6.6 and above, run the following command:

java -jar oak-run-*.jar datastorecheck --consistency -ref --id --fds crx-quickstart/install/org.apache.jackrabbit.oak.plugins.blob.datastore.FileDataStore.config --repoHome crx-quickstart/repository --store crx-quickstart/repository/datastore --dump temp --verbose --track

Create a list of paths of the missing files.

The next step in recovering the missing files is to compile a complete list of the paths of the files that are missing.

Search the consistency check output for all occurrences of error Record not Found

a. In Linux or Unix: Use this command to output the list of missing files to a file missing_ds_files.txt:

grep "blobId" consistency_check_output.txt | grep -Eo "[ 0-9a-f] {40,200}" | awk '{ print
substr($1, 0,2) "/" substr($1, 3,2) "/" substr($1, 5,2) "/" $1 }' | sort -u >
missing_ds_files.txt

Note - If your Adobe Experience Manager instance is not starting up due to the blobId errors, then search your log files under crx-quickstart/logs for all occurrences of the error 'Error occurred while obtaining InputStream for blobId' instead:

grep "Error occurred while obtaining InputStream for blobId" error.log* | grep -Eo "[ 0-9a-f] {40,200}" | awk '{ print substr($1, 0,2) "/" substr($1, 3,2) "/" substr($1, 5,2) "/" $1 }' | sort -u > missing_ds_files.txt

If the command worked properly, then the contents of missing_ds_files.txt would look similar to the following:

12/92/04/129204a6dd0ce2cd5ca19c721b6f52ee2b3630e2
9f/d8/38/9fd8386d20cf55e7e0024e18d0c7d4e8400454ee
7a/13/15/7a1315788f45dafd6630454f04183601682a9f80
28/37/d2/2837d24aed3ff223cd40e90222226c4ef2e2a0c6

b. In Windows: Use a text editor such as Textpad or Notepad++ to find all occurrences of Record not found. Then, after finding all such occurrences, extract the filenames using a macro, writing a script or by manually copying and constructing the filenames in a new text file.

DataStore file paths are constructed from the record name in this format: {first two chars of record id}/{second two chars of record id}/{third two chars of record id}/{record id}

For example, in this error:

java.lang.RuntimeException: Error occurred while obtaining InputStream for blobId
[ add1dd8fc5093b27b1fae1b753cb48b24ef3231f#101983]

the record id is add1dd8fc5093b27b1fae1b753cb48b24ef3231f and the file path is ad/d1/dd/add1dd8fc5093b27b1fae1b753cb48b24ef3231f

Recover the missing files.

Now use the output from the last step to hunt down the same files in other Adobe Experience Manager instances in your environment. Since the datastore files are stored uniquely, you can copy them from other Adobe Experience Manager instances in your environment.

If you cannot find some of the files in other instances, then search your backups and restore them from there if possible.

In Linux, you could log in to each of the working Adobe Experience Manager instances and use a command like rsync to copy over any of the missing files that exist in them. For example, by running it from the datastore directory on the server that has the missing files:

rsync -avR --files-from=missing_ds_files.txt . user@hostname-of-server-missing-files:/path/to/crx-quickstart/repository/repository/datastore/

The command run an rsync that copies over any files listed in missing_ds_files.txt that exist in the server.

Clean up unrecoverable file references.

If you were not able to recover some of the files from backup or from other Adobe Experience Manager instances then clean up or fix the bad datastore references. Rerun the DataStore Consistency Check as we ran it in step 4. You get a current list of missing files.

Review each of the node paths listed which are referencing missing datastore files. Review any missing DAM assets or files uploaded to pages with your user. Have them reupload any missing ones they need. Any they don’t need can safely be deleted via the Adobe Experience Manager user interface. If anything is missing from under /var/audit or /var/eventing can safely be deleted. For any files you are not sure about, then go here and contact the AEM support team for assistance.

Reindex all async Oak indexes.

Since index files are stored within the Oak repository, it could also be possible that some files are missing from /oak:index lucene based indexees.

Those missing files would not get reported in the /system/console/repositorycheck consistency check as they are hidden from repository traversals. Unfortunately, such files would not match across different AEM instances.

To fix such inconsistencies, reindex the affected asynchronous Oak indexes.

WARNING: This operation is extremely expensive and will require extended downtime, anywhere from 10 minutes to multiple days to complete depending on the volume of content in the system.

To reindex the async indexes:

1. Download oak-run 1.0.x matching the version you have installed in your Oak environment: https://repo1.maven.org/maven2/org/apache/jackrabbit/oak-run/

2. Upload the oak-run to the Adobe Experience Manager server.

3. Stop all Adobe Experience Manager instances.

4. Run the command below corresponding to your Adobe Experience Manager instance’s Oak storage:

   TarMK command:

   code language-none
   java -Xmx4096m -jar /apps/staging/oak-run-*-*.jar checkpoints crx-quickstart/repository/segmentstore rm-all

   MongoMK command:

   code language-none
   java -Xmx4096m -jar oak-run-*-*.jar checkpoints mongodb://localhost/aem-author rm-all

5. Start Adobe Experience Manager again and monitor your log files for INFO log messages from org.apache.jackrabbit.oak.plugins.index. If you want to see more details on the indexing, then go to /system/console/slinglog UI and enable debug level logging for org.apache.jackrabbit.oak.plugins.index.

   You should see log messages like these:

   code language-none

   23.06.2015 14:26:23.070 *INFO* [ FelixStartLevel] org.apache.jackrabbit.oak.plugins.index.IndexUpdate Reindexing will be performed for following indexes: [ /oak:index/cqAcUUID, /oak:index/nodetype, /oak:index/deviceIdentificationMode, /oak:index/campaignpath, /oak:index/active, /oak:index/jcrFrozenMixinTypes] 23.06.2015 14:26:23.517 *INFO* [ FelixStartLevel] org.apache.jackrabbit.oak.plugins.index.IndexUpdate Reindexing Traversed #10000 /jcr:system/jcr:versionStorage/c8/5f ... 23.06.2015 14:28:51.999 *INFO* [ pool-8-thread-1] org.apache.jackrabbit.oak.plugins.index.IndexUpdate Indexing report     - /oak:index/counter*(708)     - /oak:index/authorizables*(159)     - /oak:index/cqPageLucene*(1913)     - /oak:index/ntBaseLucene*(444)     - /oak:index/cqTagLucene*(512)     - /oak:index/workflowDataLucene*(116) ... 23.06.2015 14:28:52.009 *INFO* [ pool-8-thread-1] org.apache.jackrabbit.oak.plugins.index.AsyncIndexUpdate Reindexing (async) completed for indexes: [ /oak:index/counter*(708), /oak:index/authorizables*(159), /oak:index/cqPageLucene*(1913), /oak:index/ntBaseLucene*(444), /oak:index/cqTagLucene*(512), /oak:index/workflowDataLucene*(116)] in 30.36 s

6. If indexing fails for some reason, then you would see it looping, repeating this log message:

   code language-none
   23.06.2015 14:26:23.070 *INFO* [ FelixStartLevel] org.apache.jackrabbit.oak.plugins.index.IndexUpdate Reindexing will be performed for following indexes: [ /oak:index/cqAcUUID, /oak:index/nodetype, /oak:index/deviceIdentificationMode, /oak:index/campaignpath, /oak:index/active, /oak:index/jcrFrozenMixinTypes]