Migration steps of an Existing AEM Datastore to AWS S3

Last update: 2023-11-21

Follow the steps below to migrate an existing Adobe Experience Manager (AEM) datastore to Amazon Simple Storage Service (AWS S3).

Description

Environment

Experience Manager

Issue/Symptoms

What are the steps for migration of an existing Adobe Experience Manager (AEM) datastore to Amazon Simple Storage Service (AWS S3).

Resolution

Below are the S3 Migration steps:

  1. Enable debug logs for packages org.apache.jackrabbit.oak.blob and org.apache.jackrabbit.oak.plugins.blob.
  2. Shutdown AEM.
  3. Backup the existing FDS configuration in …/crx-quickstart/install/.
  4. Delete the FileDataStore.config in …/crx-quickstart/install/.
  5. Copy the S3 connector to the …/crx-quickstart/install/folder.
  6. Create the S3 configuration file into …/crx-quickstart/install/. Add the below configurations:
    • accessKey=
    • connectionTimeout=“120000”
    • maxConnections=“40”
    • maxErrorRetry=“10”
    • s3Bucket=“XXXXXXXX”
    • s3Region=“XXXXXXXX”
    • secretKey=“XXXXXXXX”
    • socketTimeout=“120000”
    • writeThreads=“30”
    • path=“crx-quickstart/repository/datastore”
  7. Created folders - upload/download/tmp subdirectories inside …/crx-quickstart/repository/datastore/.
  8. Make sure the install folder and …/crx-quickstart/repository/datastore folders are owned by user with which AEM is running.
  9. Move all the contents from ../crx-quickstart/repository/datastore/ to …/crx-quickstart/repository/datastore/upload/folder.
  10. Update the runmode in startup script so that AEM starts with crx3tar-nofds runmode.
  11. Delete old AEM setting regarding startup modes (i.e., search and delete sling.options.file)
  12. Delete old AEM cached FDS config from …/crx-quickstart/launchpad/config/org/apache/jackrabbit/oak/plugins/blob/datastore/.
  13. Start AEM.

Next steps:

  1. Monitor the error.log file when AEM starts uploading files in S3 and startup is finished.
  2. Check the upload folder and see if it’s empty.
  3. Check all the bundles are started and that there is no FileDataStore configuration and only S3 configuration exists.
  4. Check if new sling.options.file is created.
  5. Logs for upload:
05.02.2020 20:19:53.326 *INFO* [ oak-ds-async-upload-thread-5]  org.apache.jackrabbit.oak.plugins.blob.UploadStagingCache Successfully added [ ffc3b119d6b8dc9955edadbf0eca207d9cce2937cf97abc50accbc7f2cb342e5] , [ C:\AEM\6.4\6.4.6.0\Latest_Correct\crx-quickstart\repository\datastore\upload\ff\c3\b1\ffc3b119d6b8dc9955edadbf0eca207d9cce2937cf97abc50accbc7f2cb342e5]
05.02.2020 20:19:53.398 *INFO* [ oak-ds-async-upload-thread-8]  org.apache.jackrabbit.oak.plugins.blob.UploadStagingCache Successfully added [ ffdc13dce5fd816e1a63e83ecec1cba009024a4f7e4f4297f648e8cf3a28f98b] , [ C:\AEM\6.4\6.4.6.0\Latest_Correct\crx-quickstart\repository\datastore\upload\ff\dc\13\ffdc13dce5fd816e1a63e83ecec1cba009024a4f7e4f4297f648e8cf3a28f98b]
05.02.2020 20:19:53.451 *INFO* [ oak-ds-async-upload-thread-6]  org.apache.jackrabbit.oak.plugins.blob.UploadStagingCache Successfully added [ repository-151f8454-c398-4d12-b5e8-1617aa8cab79] , [ C:\AEM\6.4\6.4.6.0\Latest_Correct\crx-quickstart\repository\datastore\upload\re\po\si\repository-151f8454-c398-4d12-b5e8-1617aa8cab79]
05.02.2020 20:19:53.454 *INFO* [ oak-ds-async-upload-thread-3]  org.apache.jackrabbit.oak.plugins.blob.UploadStagingCache Successfully added [ fff3fcbb484d002d7db9cccc96971c06e8926a4a5e07cf93f4db7c450690fa71] , [ C:\AEM\6.4\6.4.6.0\Latest_Correct\crx-quickstart\repository\datastore\upload\ff\f3\fc\fff3fcbb484d002d7db9cccc96971c06e8926a4a5e07cf93f4db7c450690fa71]
05.02.2020 20:19:53.488 *INFO* [ oak-ds-async-upload-thread-4]  org.apache.jackrabbit.oak.plugins.blob.UploadStagingCache Successfully added [ ffbbcd9f72eb5373761751e8d5ecbf4022f58e70119e354557bc32eb17fe6b2a] , [ C:\AEM\6.4\6.4.6.0\Latest_Correct\crx-quickstart\repository\datastore\upload\ff\bb\cd\ffbbcd9f72eb5373761751e8d5ecbf4022f58e70119e354557bc32eb17fe6b2a]

Deletion of upload files:

05.02.2020 20:22:40.794 *INFO* [ oak-ds-cache-scheduled-thread-1]  org.apache.jackrabbit.oak.plugins.blob.UploadStagingCache Starting purge of uploaded files
05.02.2020 20:22:40.795 *INFO* [ oak-ds-cache-scheduled-thread-1]  org.apache.jackrabbit.oak.plugins.blob.DataStoreCacheUtils Deleted file [ C:\AEM\6.4\6.4.6.0\Latest_Correct\crx-quickstart\repository\datastore\upload\bc\18\5a\bc185acc8571b7e0f97dac92b0285fe248004909c3d8264e03cfb2a8101bada6]
05.02.2020 20:22:40.800 *INFO* [ oak-ds-cache-scheduled-thread-1]  org.apache.jackrabbit.oak.plugins.blob.DataStoreCacheUtils Deleted file [ C:\AEM\6.4\6.4.6.0\Latest_Correct\crx-quickstart\repository\datastore\upload\fa\6f\c1\fa6fc18d5b95cf213ff5ac5d9eb0fed7c61310ac2c373ca2cbf187844bf39c24]
05.02.2020 20:22:40.801 *INFO* [ oak-ds-cache-scheduled-thread-1]  org.apache.jackrabbit.oak.plugins.blob.DataStoreCacheUtils Deleted file [ C:\AEM\6.4\6.4.6.0\Latest_Correct\crx-quickstart\repository\datastore\upload\d9\b4\f3\d9b4f3f58a9b83a1e1647cc23b77d672836171afdccbbbd8726f480b741a4c2e]
05.02.2020 20:22:40.802 *INFO* [ oak-ds-cache-scheduled-thread-1]  org.apache.jackrabbit.oak.plugins.blob.DataStoreCacheUtils Deleted file [ C:\AEM\6.4\6.4.6.0\Latest_Correct\crx-quickstart\repository\datastore\upload\c6\3a\df\c63adf50a7ebc7b1652740fb8be9b72f5b76d22477f0d411becab2f8eeceb70b]
Previous page
Next page

On this page