Performance tuning for S3 Datastore - Asynchronous mode

Description description


Having an external S3 Datastore.

Performing lots of intensive  upload  activities:

  • multiple and large assets ingestions
  • importing large index definition (generated out-of-the-band)

By default, S3 uploads are  synchronous  and mono-threaded, therefore huge operations can take some time to perform

Resolution resolution

The documentation specifies, however, that S3 uploads can be configured to perform asynchronously  and multi-threaded (10 threads by default):

In order to greatly improve uploads time, it is possible to enable and test the asynchronous mode by adding a Java System parameter “oak.lucene.ds.async” to the startup command line:


To change the number of upload threads, you need to specify the  uploadThreads parameter in your S3Datastore.config file.

These settings need to be tested first in a lower environment, performing some benchmarks, before pushing the change to production.