Documentation

Steps for migration of an existing AEM datastore to AWS S3

Last update: Wed May 15 2024 00:00:00 GMT+0000 (Coordinated Universal Time)

This article will help you learn about the steps required to migrate an existing Adobe Experience Manager (AEM) datastore to Amazon Simple Storage Service (AWS S3).

Description description

Environment

Adobe Experience Manager

Issue/Symptoms

What are the steps for migration of an existing Adobe Experience Manager (AEM) datastore to Amazon Simple Storage Service (AWS S3)?

Resolution resolution

Below are the steps for migration of an existing AEM datastore to AWS S3:

  1. Enable debug logs for the packages org.apache.jackrabbit.oak.blob  and org.apache.jackrabbit.oak.plugins.blob.

  2. Shutdown AEM.

  3. Back up the existing FDS configuration in …/crx-quickstart/install/.

  4. Delete the FileDataStore.config in …/crx-quickstart/install/.

  5. Copy the S3 connector to the …/crx-quickstart/install/folder.

  6. Create the S3 configuration file into …/crx-quickstart/install/.

  7. Add the below configurations:

    • accessKey=
    • connectionTimeout=“120000”
    • maxConnections=“40”
    • maxErrorRetry=“10”
    • s3Bucket=“XXXXXXXX”
    • s3Region=“XXXXXXXX”
    • secretKey=“XXXXXXXX”
    • socketTimeout=“120000”
    • writeThreads=“30”
    • path=“crx-quickstart/repository/datastore”

  8. Create folders - upload/download/tmp subdirectories inside …/crx-quickstart/repository/datastore/.

  9. Make sure the install folder and …/crx-quickstart/repository/datastore folders are owned by user with which AEM is running.

  10. Move all the contents from ../crx-quickstart/repository/datastore/ to …/crx-quickstart/repository/datastore/upload/folder.

  11. Update the runmode in startup script so that AEM starts with crx3tar-nofds runmode.

  12. Delete the old AEM setting regarding startup modes (i.e., search and delete sling.options.file)

  13. Delete old AEM cached FDS config from …/crx-quickstart/launchpad/config/org/apache/jackrabbit/oak/plugins/blob/datastore/.

  14. Start AEM.

Next steps:

  1. Monitor the error.log file when AEM starts uploading files in S3 and the startup is finished.
  2. Check the upload folder and see if it’s empty.
  3. Check all the bundles are started and that there is no FileDataStore configuration and only S3 configuration exists.
  4. Check if a new sling.options.file is created.
  5. Logs for upload:
05.02.2020 20:19:53.326 *INFO* [ oak-ds-async-upload-thread-5]  org.apache.jackrabbit.oak.plugins.blob.UploadStagingCache Successfully added [ ffc3b119d6b8dc9955edadbf0eca207d9cce2937cf97abc50accbc7f2cb342e5] , [ C:\AEM\6.4\6.4.6.0\Latest_Correct\crx-quickstart\repository\datastore\upload\ff\c3\b1\ffc3b119d6b8dc9955edadbf0eca207d9cce2937cf97abc50accbc7f2cb342e5]
05.02.2020 20:19:53.398 *INFO* [ oak-ds-async-upload-thread-8]  org.apache.jackrabbit.oak.plugins.blob.UploadStagingCache Successfully added [ ffdc13dce5fd816e1a63e83ecec1cba009024a4f7e4f4297f648e8cf3a28f98b] , [ C:\AEM\6.4\6.4.6.0\Latest_Correct\crx-quickstart\repository\datastore\upload\ff\dc\13\ffdc13dce5fd816e1a63e83ecec1cba009024a4f7e4f4297f648e8cf3a28f98b]
05.02.2020 20:19:53.451 *INFO* [ oak-ds-async-upload-thread-6]  org.apache.jackrabbit.oak.plugins.blob.UploadStagingCache Successfully added [ repository-151f8454-c398-4d12-b5e8-1617aa8cab79] , [ C:\AEM\6.4\6.4.6.0\Latest_Correct\crx-quickstart\repository\datastore\upload\re\po\si\repository-151f8454-c398-4d12-b5e8-1617aa8cab79]
05.02.2020 20:19:53.454 *INFO* [ oak-ds-async-upload-thread-3]  org.apache.jackrabbit.oak.plugins.blob.UploadStagingCache Successfully added [ fff3fcbb484d002d7db9cccc96971c06e8926a4a5e07cf93f4db7c450690fa71] , [ C:\AEM\6.4\6.4.6.0\Latest_Correct\crx-quickstart\repository\datastore\upload\ff\f3\fc\fff3fcbb484d002d7db9cccc96971c06e8926a4a5e07cf93f4db7c450690fa71]
05.02.2020 20:19:53.488 *INFO* [ oak-ds-async-upload-thread-4]  org.apache.jackrabbit.oak.plugins.blob.UploadStagingCache Successfully added [ ffbbcd9f72eb5373761751e8d5ecbf4022f58e70119e354557bc32eb17fe6b2a] , [ C:\AEM\6.4\6.4.6.0\Latest_Correct\crx-quickstart\repository\datastore\upload\ff\bb\cd\ffbbcd9f72eb5373761751e8d5ecbf4022f58e70119e354557bc32eb17fe6b2a]

Deletion of upload files:

05.02.2020 20:22:40.794 *INFO* [ oak-ds-cache-scheduled-thread-1]  org.apache.jackrabbit.oak.plugins.blob.UploadStagingCache Starting purge of uploaded files
05.02.2020 20:22:40.795 *INFO* [ oak-ds-cache-scheduled-thread-1]  org.apache.jackrabbit.oak.plugins.blob.DataStoreCacheUtils Deleted file [ C:\AEM\6.4\6.4.6.0\Latest_Correct\crx-quickstart\repository\datastore\upload\bc\18\5a\bc185acc8571b7e0f97dac92b0285fe248004909c3d8264e03cfb2a8101bada6]
05.02.2020 20:22:40.800 *INFO* [ oak-ds-cache-scheduled-thread-1]  org.apache.jackrabbit.oak.plugins.blob.DataStoreCacheUtils Deleted file [ C:\AEM\6.4\6.4.6.0\Latest_Correct\crx-quickstart\repository\datastore\upload\fa\6f\c1\fa6fc18d5b95cf213ff5ac5d9eb0fed7c61310ac2c373ca2cbf187844bf39c24]
05.02.2020 20:22:40.801 *INFO* [ oak-ds-cache-scheduled-thread-1]  org.apache.jackrabbit.oak.plugins.blob.DataStoreCacheUtils Deleted file [ C:\AEM\6.4\6.4.6.0\Latest_Correct\crx-quickstart\repository\datastore\upload\d9\b4\f3\d9b4f3f58a9b83a1e1647cc23b77d672836171afdccbbbd8726f480b741a4c2e]
05.02.2020 20:22:40.802 *INFO* [ oak-ds-cache-scheduled-thread-1]  org.apache.jackrabbit.oak.plugins.blob.DataStoreCacheUtils Deleted file [ C:\AEM\6.4\6.4.6.0\Latest_Correct\crx-quickstart\repository\datastore\upload\c6\3a\df\c63adf50a7ebc7b1652740fb8be9b72f5b76d22477f0d411becab2f8eeceb70b]
recommendation-more-help
3d58f420-19b5-47a0-a122-5c9dab55ec7f