AEM author instability and severe latency caused by inadequate EBS storage performance (AMS)

This article describes recurring performance degradation and instability on AEM Author instances hosted on Adobe Managed Services (AMS), where the underlying EBS volume type does not meet the I/O requirements of AEM’s TarMK repository.

Description description

Environment

Adobe Experience Manager deployments using TarMK repositories on Adobe Managed Services (AMS) backed by Amazon EBS storage.

Issue/Symptoms

The AEM Author instance experiences recurring instability under normal authoring workloads, including one or more of the following:

  • Severe latency in the Author UI, with response times of one minute or more for standard editor, shell, and CSRF token requests.
  • 504 gateway timeouts and 404 errors reported by authors.
  • Failed or significantly delayed publish operations.
  • Remote DAM event processing falling behind and becoming blocked.

The following warnings and errors may appear in error.log during or after degraded periods:

*WARN* [ sling-oak-N-org.apache.jackrabbit.oak.plugins.index.AsyncIndexUpdate-async]
org.apache.jackrabbit.oak.segment.scheduler.LockBasedScheduler
Failed to create checkpoint <uuid> in 10 seconds.

*WARN* [ oak-lucene-N]
org.apache.jackrabbit.oak.plugins.index.lucene.directory.CopyOnReadDirectory
Couldn't compute safe timestamp to delete files from NIOFSDirectory@/mnt/crx/.../index/.../data

*WARN* [ oak-lucene-N]
org.apache.jackrabbit.oak.plugins.index.lucene.IndexCopier
File <segment> doesn't exist in /mnt/crx/.../repository/index/.../data

*WARN* [ sling-default-N-Registered Service.NNNNN]
com.adobe.cq.remotedam.internal.lifecycle.event.consumer.impl.LifecycleEventConsumerImpl
This sites instance is lagging far too behind Remote DAM. Pulling events is blocked

Correlation of request.log against these periods typically reveals clusters of requests completing after 30 seconds to several minutes across health check, editor, pulse, and CSRF token endpoints simultaneously. This pattern indicates a JVM-wide stall rather than a single slow request.

Cause

AEM’s TarMK repository, stored under /mnt/crx, performs frequent sequential and random I/O operations for segment store writes, checkpoint creation, Lucene index updates, and compaction. These repository workloads require consistent storage latency and sufficient IOPS.

Standard magnetic EBS volumes may not provide adequate performance characteristics for these workloads. When storage throughput becomes constrained, the operating system can experience sustained I/O wait, causing multiple JVM thread pools to stall simultaneously and resulting in broad application latency across unrelated requests.

This issue is most commonly observed in non-production AMS environments that were originally provisioned with magnetic EBS volumes before gp3 EBS volumes became the standard repository storage option. Production environments are often unaffected because they were provisioned on, or later migrated to, gp3 storage.

Resolution resolution

To identify and resolve the recurring AEM Author instability, severe latency, and repository performance issues, follow these steps:

  1. Verify the Amazon EBS volume type backing /mnt/crx on the affected Author instance. Contact your Customer Success Engineer (CSE) to confirm the volume configuration and determine whether the repository is hosted on a gp3 EBS volume.
  2. Evaluate a migration to a gp3 EBS volume if the repository is hosted on standard magnetic storage. Review any service or cost considerations with your CSE based on your AMS agreement.
  3. Request a storage migration through your CSE. Coordinate with the AMS infrastructure team to create the migration plan and confirm the target IOPS and throughput settings based on repository size and workload requirements.
  4. Migrate the repository volume to a gp3 EBS volume. The AMS infrastructure team typically performs this process by stopping the AEM instance, creating a snapshot of the existing volume, migrating the storage, and restarting the instance.
  5. Include affected Publisher instances in the migration plan if they use the same storage configuration.
  6. Measure system performance after the migration. Verify that I/O wait, swap usage, and load average have decreased during typical authoring activity.
  7. Review request.log and confirm that editor and utility endpoints no longer exhibit multi-second or minute-long response times.
  8. Review error.log and verify that checkpoint failure messages and Lucene index warnings do not recur.
  9. Validate that the Author page editor loads normally, publish operations complete successfully, async indexing remains current, and Remote DAM events process without lag.
  10. Correlate recurring performance degradation with repository-intensive activities such as large replication bursts, MSM rollouts, package deployments, or large-scale asset ingestion if instability persists after the migration.
  11. Adjust scheduling or throttling of high-impact workloads when peak repository demand continues to affect performance.
recommendation-more-help
experience-cloud-kcs-help-kbarticles