Environment
Issue/Symptoms
Error occurred while obtaining InputStream
for blobId
You have a FileDataStore configured in your Adobe Experience Manager 6.x / Oak 1.x system, and in the error.log you see “Error occurred while obtaining InputStream for blobId.”
1 2 |
10.09.2015 10:38:04.220 *ERROR* pool-9-thread-3 org.apache.sling.commons.scheduler.impl.QuartzScheduler Exception during job execution of org.apache.jackrabbit.oak.plugins.index.AsyncIndexUpdate@7fe7e8fa : Error occurred while obtaining InputStream for blobId add1dd8fc5093b27b1fae1b753cb48b24ef3231f#101983``java.lang.RuntimeException: Error occurred while obtaining InputStream for blobId add1dd8fc5093b27b1fae1b753cb48b24ef3231f#101983 |
---|
This error means that you are possibly missing files from your Adobe Experience Manager datastore directory. The loss of datastore files can be due to a failure of Oak Blob Garbage Collection, a disk space outage, disk, or network share instability. Or, it could be due to a user erroneously deleting files from the server.
To recover the missing files, follow the steps below.
1. Install the Latest CFP/SP
You can find the latest hot fixes on Adobe Package Share.
2. Run Datastore Consistency Check
Using oak-run jar 1.6.6 and above, run the following command:
java -jar oak-run-*.jar datastorecheck --consistency -ref --id --fds crx-quickstart/install/org.apache.jackrabbit.oak.plugins.blob.datastore.FileDataStore.config --repoHome crx-quickstart/repository --store crx-quickstart/repository/datastore --dump temp --verbose --track
3. Create a list of paths of the missing files
The next step in recovering the missing files is to compile a complete list of the paths of the files that are missing.
a. In Linux or Unix: Use this command to output the list of missing files to a file missing_ds_files.txt:
| grep
"blobId"
consistency_check_output.txt | ``grep
-Eo ``"0-9a-f{40,200}"
| ``awk
'{ print substr($1, 0,2) "/" substr($1, 3,2) "/" substr($1, 5,2) "/" $1 }'
| ``sort
-u missing_ds_files.txt
|
| — |
| grep
"Error occurred while obtaining InputStream for blobId"
error.log* | ``grep
-Eo ``"0-9a-f{40,200}"
| ``awk
'{ print substr($1, 0,2) "/" substr($1, 3,2) "/" substr($1, 5,2) "/" $1 }'
| ``sort
-u missing_ds_files.txt
|
| — |
12/92/04/129204a6dd0ce2cd5ca19c721b6f52ee2b3630e2``9f/d8/38/9fd8386d20cf55e7e0024e18d0c7d4e8400454ee``7a/13/15/7a1315788f45dafd6630454f04183601682a9f80``28/37/d2/2837d24aed3ff223cd40e90222226c4ef2e2a0c6 |
---|
DataStore file paths are constructed from the record name in this format:
{first two chars of record id}/{second two chars of record id}/{third two chars of record id}/{record id}
For example, in this error:
java.lang.RuntimeException: Error occurred while obtaining InputStream for blobId add1dd8fc5093b27b1fae1b753cb48b24ef3231f#101983 |
---|
If you cannot find some of the files in other instances, then search your backups and restore them from there if possible.
In Linux, you could log in to each of the working Adobe Experience Manager instances and use a command like rsync to copy over any of the missing files that exist in them. For example, by running it from the datastore directory on the server that has the missing files:
rsync -avR --files-from=missing_ds_files.txt . user@``hostname``-of-server-missing-files:``/path/to/crx-quickstart/repository/repository/datastore/ |
---|
Review each of the node paths listed which are referencing missing datastore files. Review any missing DAM assets or files uploaded to pages with your user. Have them reupload any missing ones they need. Any they don’t need can safely be deleted via the Adobe Experience Manager user interface. If anything is missing from under /var/audit or /var/eventing can safely be deleted. For any files you are not sure about, then go here and contact the AEM support team for assistance.
6. Reindex all async Oak indexes
Since index files are stored within the Oak repository, it could also be possible that some files are missing from /oak:index lucene based indexees.
Those missing files would not get reported in the /system/console/repositorycheck consistency check as they are hidden from repository traversals. Unfortunately, such files would not match across different AEM instances.
To fix such inconsistencies, reindex the affected asynchronous Oak indexes.
WARNING: This operation is extremely expensive and will require extended downtime, anywhere from 10 minutes to multiple days to complete depending on the volume of content in the system.
To reindex the async indexes:
Download oak-run 1.0.x matching the version you have installed in your Oak environment: https://repo1.maven.org/maven2/org/apache/jackrabbit/oak-run/
Upload the oak-run to the Adobe Experience Manager server.
Stop all Adobe Experience Manager instances.
Run the command below corresponding to your Adobe Experience Manager instance’s Oak storage:
TarMK command:
java -Xmx4096m -jar /apps/staging/oak-run-*-*.jar checkpoints crx-quickstart/repository/segmentstore rm-all |
---|
1 | java -Xmx4096m -jar oak-run-*-*.jar checkpoints mongodb://localhost/aem-author rm-all |
---|
You should see log messages like these:
23.06.2015 14:26:23.070 *INFO* FelixStartLevel org.apache.jackrabbit.oak.plugins.index.IndexUpdate Reindexing will be performed for following indexes: /oak:index/cqAcUUID, /oak:index/nodetype, /oak:index/deviceIdentificationMode, /oak:index/campaignpath, /oak:index/active, /oak:index/jcrFrozenMixinTypes 23.06.2015 14:26:23.517 *INFO* FelixStartLevel org.apache.jackrabbit.oak.plugins.index.IndexUpdate Reindexing Traversed #10000 /jcr:system/jcr:versionStorage/c8/5f``...``23.06.2015 14:28:51.999 *INFO* pool-8-thread-1 org.apache.jackrabbit.oak.plugins.index.IndexUpdate Indexing report`` ``- /oak:index/counter*(708)`` ``- /oak:index/authorizables*(159)`` ``- /oak:index/cqPageLucene*(1913)`` ``- /oak:index/ntBaseLucene*(444)`` ``- /oak:index/cqTagLucene*(512)`` ``- /oak:index/workflowDataLucene*(116)``... 23.06.2015 14:28:52.009 *INFO* pool-8-thread-1 org.apache.jackrabbit.oak.plugins.index.AsyncIndexUpdate Reindexing (async) completed for indexes: /oak:index/counter*(708), /oak:index/authorizables*(159), /oak:index/cqPageLucene*(1913), /oak:index/ntBaseLucene*(444), /oak:index/cqTagLucene*(512), /oak:index/workflowDataLucene*(116) in 30.36 s |
---|
23.06.2015 14:26:23.070 *INFO* FelixStartLevel org.apache.jackrabbit.oak.plugins.index.IndexUpdate Reindexing will be performed for following indexes: /oak:index/cqAcUUID, /oak:index/nodetype, /oak:index/deviceIdentificationMode, /oak:index/campaignpath, /oak:index/active, /oak:index/jcrFrozenMixinTypes |
---|