Migration steps of an Existing AEM Datastore to AWS S3

Description

This article is related to issues while performing migration of an existing AEM datastore to AWS S3.

Resolution

Below are the S3 migration steps:

1-Enable debug logs for packages 'org.apache.jackrabbit.oak.blob' and 'org.apache.jackrabbit.oak.plugins.blob'.

2-Shutdown AEM.

3-Backup the existing FDS configuration in ../crx-quickstart/install/.

4-Delete the FileDataStore.config in ../crx-quickstart/install/.

5-Copy the S3 connector to the ../crx-quickstart/install/ folder.

6-Create the S3 configuration file into ../crx-quickstart/install/. Add the below configurations:

accessKey=
connectionTimeout="120000"
maxConnections="40"
maxErrorRetry="10"
s3Bucket="XXXXXXXX"
s3Region="XXXXXXXX"
secretKey="XXXXXXXX"
socketTimeout="120000"
writeThreads="30"
path="crx-quickstart/repository/datastore"

7-Created folders - upload/download/tmp subdirectories inside ../crx-quickstart/repository/datastore/.

8-Make sure the install folder and ../crx-quickstart/repository/datastore folders are owned by user with which AEM is running.

9-Move all the contents from ../crx-quickstart/repository/datastore/ to ../crx-quickstart/repository/datastore/upload/ folder.

10-Update the runmode in startup script so that AEM starts with crx3tar-nofds runmode.

11-Delete old AEM setting regarding startup modes (i.e. search and delete sling.options.file)

12-Delete old AEM cached FDS config from ../crx-quickstart/launchpad/config/org/apache/jackrabbit/oak/plugins/blob/datastore/.

13-Start AEM.


Next steps:

1-Monitor the error.log file when AEM starts uploading files in S3 and startup is finished.

2-Check the upload folder and see if it's empty.

3-Check all the bundles are started and that there no FileDataStore config and only S3 configuration exists.

4-Check if new sling.options.file is created.

Logs for upload

05.02.2020 20:19:53.326 *INFO* oak-ds-async-upload-thread-5 org.apache.jackrabbit.oak.plugins.blob.UploadStagingCache Successfully added ffc3b119d6b8dc9955edadbf0eca207d9cce2937cf97abc50accbc7f2cb342e5, C:\AEM\6.4\6.4.6.0\Latest_Correct\crx-quickstart\repository\datastore\upload\ff\c3\b1\ffc3b119d6b8dc9955edadbf0eca207d9cce2937cf97abc50accbc7f2cb342e5
05.02.2020 20:19:53.398 *INFO* oak-ds-async-upload-thread-8 org.apache.jackrabbit.oak.plugins.blob.UploadStagingCache Successfully added ffdc13dce5fd816e1a63e83ecec1cba009024a4f7e4f4297f648e8cf3a28f98b, C:\AEM\6.4\6.4.6.0\Latest_Correct\crx-quickstart\repository\datastore\upload\ff\dc\13\ffdc13dce5fd816e1a63e83ecec1cba009024a4f7e4f4297f648e8cf3a28f98b
05.02.2020 20:19:53.451 *INFO* oak-ds-async-upload-thread-6 org.apache.jackrabbit.oak.plugins.blob.UploadStagingCache Successfully added repository-151f8454-c398-4d12-b5e8-1617aa8cab79, C:\AEM\6.4\6.4.6.0\Latest_Correct\crx-quickstart\repository\datastore\upload\re\po\si\repository-151f8454-c398-4d12-b5e8-1617aa8cab79
05.02.2020 20:19:53.454 *INFO* oak-ds-async-upload-thread-3 org.apache.jackrabbit.oak.plugins.blob.UploadStagingCache Successfully added fff3fcbb484d002d7db9cccc96971c06e8926a4a5e07cf93f4db7c450690fa71, C:\AEM\6.4\6.4.6.0\Latest_Correct\crx-quickstart\repository\datastore\upload\ff\f3\fc\fff3fcbb484d002d7db9cccc96971c06e8926a4a5e07cf93f4db7c450690fa71
05.02.2020 20:19:53.488 *INFO* oak-ds-async-upload-thread-4 org.apache.jackrabbit.oak.plugins.blob.UploadStagingCache Successfully added ffbbcd9f72eb5373761751e8d5ecbf4022f58e70119e354557bc32eb17fe6b2a, C:\AEM\6.4\6.4.6.0\Latest_Correct\crx-quickstart\repository\datastore\upload\ff\bb\cd\ffbbcd9f72eb5373761751e8d5ecbf4022f58e70119e354557bc32eb17fe6b2a

Deletion of upload files

05.02.2020 20:22:40.794 *INFO* oak-ds-cache-scheduled-thread-1 org.apache.jackrabbit.oak.plugins.blob.UploadStagingCache Starting purge of uploaded files
05.02.2020 20:22:40.795 *INFO* oak-ds-cache-scheduled-thread-1 org.apache.jackrabbit.oak.plugins.blob.DataStoreCacheUtils Deleted file C:\AEM\6.4\6.4.6.0\Latest_Correct\crx-quickstart\repository\datastore\upload\bc\18\5a\bc185acc8571b7e0f97dac92b0285fe248004909c3d8264e03cfb2a8101bada6
05.02.2020 20:22:40.800 *INFO* oak-ds-cache-scheduled-thread-1 org.apache.jackrabbit.oak.plugins.blob.DataStoreCacheUtils Deleted file C:\AEM\6.4\6.4.6.0\Latest_Correct\crx-quickstart\repository\datastore\upload\fa\6f\c1\fa6fc18d5b95cf213ff5ac5d9eb0fed7c61310ac2c373ca2cbf187844bf39c24
05.02.2020 20:22:40.801 *INFO* oak-ds-cache-scheduled-thread-1 org.apache.jackrabbit.oak.plugins.blob.DataStoreCacheUtils Deleted file C:\AEM\6.4\6.4.6.0\Latest_Correct\crx-quickstart\repository\datastore\upload\d9\b4\f3\d9b4f3f58a9b83a1e1647cc23b77d672836171afdccbbbd8726f480b741a4c2e
05.02.2020 20:22:40.802 *INFO* oak-ds-cache-scheduled-thread-1 org.apache.jackrabbit.oak.plugins.blob.DataStoreCacheUtils Deleted file C:\AEM\6.4\6.4.6.0\Latest_Correct\crx-quickstart\repository\datastore\upload\c6\3a\df\c63adf50a7ebc7b1652740fb8be9b72f5b76d22477f0d411becab2f8eeceb70b

On this page