Performance tuning for S3 Datastore - Asynchronous mode

Description

Use-Case:

Having an external S3 Datastore.

Performing lots of intensive upload activities:

  • multiple and large assets ingestions
  • importing large index definition (generated out-of-the-band)

By default, S3 uploads are synchronous and mono-threaded, therefore huge operations can take some time to perform

Resolution

The documentation specifies, however, that S3 uploads can be configured to perform asynchronously and multi-threaded (10 threads by default):

https://experienceleague.adobe.com/docs/experience-manager-65/deploying/deploying/data-store-config.html?lang=en#async-upload

In order to greatly improve uploads time, it is possible to enable and test the asynchronous mode by adding a Java System parameter “oak.lucene.ds.async” to the startup command line:

-Doak.lucene.ds.async=true

To change the number of upload threads, you need to specify the uploadThreadsparameter in your S3Datastore.config file.

These settings need to be tested first in a lower environment, performing some benchmarks, before pushing the change to production.

On this page