Performance tuning for S3 Datastore - Asynchronous mode

Description description

Use-Case:

Having an external S3 Datastore.

Performing lots of intensive  upload  activities:

  • multiple and large assets ingestions
  • importing large index definition (generated out-of-the-band)

By default, S3 uploads are  synchronous  and mono-threaded, therefore huge operations can take some time to perform

Resolution resolution

The documentation specifies, however, that S3 uploads can be configured to perform asynchronously  and multi-threaded (10 threads by default):

https://experienceleague.adobe.com/docs/experience-manager-65/deploying/deploying/data-store-config.html?lang=en#async-upload

In order to greatly improve uploads time, it is possible to enable and test the asynchronous mode by adding a Java System parameter “oak.lucene.ds.async” to the startup command line:

-Doak.lucene.ds.async=true

To change the number of upload threads, you need to specify the  uploadThreads parameter in your S3Datastore.config file.

These settings need to be tested first in a lower environment, performing some benchmarks, before pushing the change to production.

recommendation-more-help
3d58f420-19b5-47a0-a122-5c9dab55ec7f