Performance tuning for S3 Datastore - Asynchronous mode
Discover the steps to configure S3 uploads for asynchronous and multi-threaded operations, with a default setting of 10 threads, in Adobe Experience Manager. To modify the number of upload threads, specify the uploadThreads parameter in the S3Datastore.config file.
Description description
Environment
Adobe Experience Manager 6.x
Issue/Symptoms
How to set S3 Datastore for asynchornous mode?
Use-Case:
Having an external S3 Datastore.
Performing lots of intensive upload activities:
- multiple and large assets ingestions
- importing large index definition (generated out-of-the-band)
By default, S3 uploads are synchronous and mono-threaded, therefore huge operations can take some time to perform
Resolution resolution
The documentation specifies, however, that S3 uploads can be configured to perform asynchronously and multi-threaded (10 threads by default):
In order to greatly improve uploads time, it is possible to enable and test the asynchronous mode by adding a Java System parameter “oak.lucene.ds.async” to the startup command line:
-Doak.lucene.ds.async=true
To change the number of upload threads, you need to specify the uploadThreads parameter in your S3Datastore.config file.
These settings need to be tested first in a lower environment, performing some benchmarks, before pushing the change to production.