Migration steps of an Existing AEM Datastore to AWS S3

Description

Environment

Experience Manager

Issue/Symptoms
What are the steps for migration of an existing Adobe Experience Manager (AEM) datastore to Amazon Simple Storage Service (AWS S3).

Resolution

Below are the S3 Migration steps:
1-Enable debug logs for packages 'org.apache.jackrabbit.oak.blob' and 'org.apache.jackrabbit.oak.plugins.blob'.

2-Shutdown AEM.

3-Backup the existing FDS configuration in ../crx-quickstart/install/.

4-Delete the FileDataStore.config in ../crx-quickstart/install/.

5-Copy the S3 connector to the ../crx-quickstart/install/folder.

6-Create the S3 configuration file into ../crx-quickstart/install/. Add the below configurations:

accessKey=
connectionTimeout=“120000”
maxConnections=“40”
maxErrorRetry=“10”
s3Bucket=“XXXXXXXX”
s3Region=“XXXXXXXX”
secretKey=“XXXXXXXX”
socketTimeout=“120000”
writeThreads=“30”
path=“crx-quickstart/repository/datastore”

7-Created folders - upload/download/tmp subdirectories inside ../crx-quickstart/repository/datastore/.

8-Make sure the install folder and ../crx-quickstart/repository/datastore folders are owned by user with which AEM is running.

9-Move all the contents from ../crx-quickstart/repository/datastore/ to ../crx-quickstart/repository/datastore/upload/folder.

10-Update the runmode in startup script so that AEM starts with crx3tar-nofds runmode.

11-Delete old AEM setting regarding startup modes (i.e. search and delete sling.options.file)

12-Delete old AEM cached FDS config from ../crx-quickstart/launchpad/config/org/apache/jackrabbit/oak/plugins/blob/datastore/.

13-Start AEM.

Next steps:

1-Monitor the error.log file when AEM starts uploading files in S3 and startup is finished.

2-Check the upload folder and see if it’s empty.

3-Check all the bundles are started and that there is no FileDataStore configuration and only S3 configuration exists.

4-Check if new sling.options.file is created.

Logs for upload

05.02.2020 20:19:53.326 *INFO* oak-ds-async-upload-thread-5 org.apache.jackrabbit.oak.plugins.blob.UploadStagingCache Successfully added ffc3b119d6b8dc9955edadbf0eca207d9cce2937cf97abc50accbc7f2cb342e5, C:\AEM\6.4\6.4.6.0\Latest_Correct\crx-quickstart\repository\datastore\upload\ff\c3\b1\ffc3b119d6b8dc9955edadbf0eca207d9cce2937cf97abc50accbc7f2cb342e5
05.02.2020 20:19:53.398 *INFO* oak-ds-async-upload-thread-8 org.apache.jackrabbit.oak.plugins.blob.UploadStagingCache Successfully added ffdc13dce5fd816e1a63e83ecec1cba009024a4f7e4f4297f648e8cf3a28f98b, C:\AEM\6.4\6.4.6.0\Latest_Correct\crx-quickstart\repository\datastore\upload\ff\dc\13\ffdc13dce5fd816e1a63e83ecec1cba009024a4f7e4f4297f648e8cf3a28f98b
05.02.2020 20:19:53.451 *INFO* oak-ds-async-upload-thread-6 org.apache.jackrabbit.oak.plugins.blob.UploadStagingCache Successfully added repository-151f8454-c398-4d12-b5e8-1617aa8cab79, C:\AEM\6.4\6.4.6.0\Latest_Correct\crx-quickstart\repository\datastore\upload\re\po\si\repository-151f8454-c398-4d12-b5e8-1617aa8cab79
05.02.2020 20:19:53.454 *INFO* oak-ds-async-upload-thread-3 org.apache.jackrabbit.oak.plugins.blob.UploadStagingCache Successfully added fff3fcbb484d002d7db9cccc96971c06e8926a4a5e07cf93f4db7c450690fa71, C:\AEM\6.4\6.4.6.0\Latest_Correct\crx-quickstart\repository\datastore\upload\ff\f3\fc\fff3fcbb484d002d7db9cccc96971c06e8926a4a5e07cf93f4db7c450690fa71
05.02.2020 20:19:53.488 *INFO* oak-ds-async-upload-thread-4 org.apache.jackrabbit.oak.plugins.blob.UploadStagingCache Successfully added ffbbcd9f72eb5373761751e8d5ecbf4022f58e70119e354557bc32eb17fe6b2a, C:\AEM\6.4\6.4.6.0\Latest_Correct\crx-quickstart\repository\datastore\upload\ff\bb\cd\ffbbcd9f72eb5373761751e8d5ecbf4022f58e70119e354557bc32eb17fe6b2a

Deletion of upload files

05.02.2020 20:22:40.794 *INFO* oak-ds-cache-scheduled-thread-1 org.apache.jackrabbit.oak.plugins.blob.UploadStagingCache Starting purge of uploaded files
05.02.2020 20:22:40.795 *INFO* oak-ds-cache-scheduled-thread-1 org.apache.jackrabbit.oak.plugins.blob.DataStoreCacheUtils Deleted file C:\AEM\6.4\6.4.6.0\Latest_Correct\crx-quickstart\repository\datastore\upload\bc\18\5a\bc185acc8571b7e0f97dac92b0285fe248004909c3d8264e03cfb2a8101bada6
05.02.2020 20:22:40.800 *INFO* oak-ds-cache-scheduled-thread-1 org.apache.jackrabbit.oak.plugins.blob.DataStoreCacheUtils Deleted file C:\AEM\6.4\6.4.6.0\Latest_Correct\crx-quickstart\repository\datastore\upload\fa\6f\c1\fa6fc18d5b95cf213ff5ac5d9eb0fed7c61310ac2c373ca2cbf187844bf39c24
05.02.2020 20:22:40.801 *INFO* oak-ds-cache-scheduled-thread-1 org.apache.jackrabbit.oak.plugins.blob.DataStoreCacheUtils Deleted file C:\AEM\6.4\6.4.6.0\Latest_Correct\crx-quickstart\repository\datastore\upload\d9\b4\f3\d9b4f3f58a9b83a1e1647cc23b77d672836171afdccbbbd8726f480b741a4c2e
05.02.2020 20:22:40.802 *INFO* oak-ds-cache-scheduled-thread-1 org.apache.jackrabbit.oak.plugins.blob.DataStoreCacheUtils Deleted file C:\AEM\6.4\6.4.6.0\Latest_Correct\crx-quickstart\repository\datastore\upload\c6\3a\df\c63adf50a7ebc7b1652740fb8be9b72f5b76d22477f0d411becab2f8eeceb70b

On this page