Resolving AEM Cold Standby synchronization failures with ReadTimeoutException

This article explains how to troubleshoot synchronization failures in Adobe Experience Manager (AEM) Cold Standby environments when the standby instance reports Failed synchronizing state and ReadTimeoutException errors. These failures occur during segment reference retrieval or Blob fetch and save operations and prevent successful synchronization between primary and standby instances. To resolve the issue, review timeout settings and validate network and storage performance to restore synchronization.

Description description

Environment

Adobe Experience Manager (AEM) with Cold Standby (all supported versions)

Issue/Symptoms

  • Standby instance logs show repeated errors similar to:

    • Failed synchronizing state
    • java.lang.IllegalStateException: Unable to read references of segment
    • io.netty.handler.timeout.ReadTimeoutException
  • Monitoring logs indicate:

    • [ ERROR] SecondsSinceLastSuccess is over 86400
    • [ ERROR] FailRequests is over 1000
  • JMX metrics at /system/console/jmx on the standby instance show:

    • SecondsSinceLastSuccess: -1
    • SyncStartTimestamp: -1
    • SyncEndTimestamp: -1
    • FailedRequests: 0
    • Status: running
    • Running: true
  • The issue may persist even after restarting or rebuilding the standby instance.

Root cause

Synchronization failures occur because of timeouts during segment reference retrieval or Blob retrieval operations. Insufficient timeout settings, network latency, or disk performance issues can interrupt data transfers between the primary and standby instances. A low standby.readtimeout value can prevent successful synchronization.

Resolution resolution

Follow the steps below to resolve the issue:

  1. Review the current timeout setting. Locate the standby instance configuration file, typically org.apache.jackrabbit.oak.segment.standby.store.StandbyStoreService.config, and check the value of standby.readtimeout. If it is set to a low value, such as 4000 ms, increase it to a higher value such as 60000 ms (60 seconds).
  2. Apply the configuration change. Save the updated configuration file and restart the standby AEM instance to apply the new timeout setting.
  3. Monitor synchronization status. Check the standby instance logs for recurring Failed synchronizing state or ReadTimeoutException errors. Review monitoring logs for messages such as [ ERROR] SecondsSinceLastSuccess is over 86400 and [ ERROR] FailRequests is over 1000. Use the JMX console (/system/console/jmx) on the standby instance to verify that metrics such as SecondsSinceLastSuccess, SyncStartTimestamp, andSyncEndTimestamp are updating as expected.
  4. Validate network and disk performance. Ensure stable network connectivity and sufficient bandwidth between the primary and standby servers. Check disk I/O performance on both servers because slow disks can contribute to timeouts.
  5. Review large segments or blobs. If large segments or blobs are present, consider increasing the timeout value further or optimizing data transfer between the instances.
  6. Verify the resolution by confirming that the standby instance resumes successful synchronization and that error counts in logs and JMX metrics decrease.
recommendation-more-help
experience-cloud-kcs-help-kbarticles