Rollout jobs are slow or stuck due to large debug queries in AEM as a Cloud Service
Adobe Experience Manager as a Cloud Service (AEMaaCS) rollout jobs can become slow or stuck when large debug search queries consume most of the JVM’s memory and CPU, leaving insufficient resources for rollout processing. The rollout mechanism continues to function but experiences significant delays because of resource contention. To resolve the issue, review debug query usage and access, monitor JVM resources, address confirmed resource contention, review group memberships, and monitor rollout processing after remediation.
Description description
Environment
Adobe Experience Manager as a Cloud Service (AEMaaCS) (all versions)
Issue/Symptoms
- Rollout jobs in the AEM Author environment remain queued for extended periods or don’t complete.
- Content launches are delayed, impacting time-sensitive business events.
- High memory and CPU usage occurs during the incident.
- The issue is reproducible and specific to the AEM Author environment and doesn’t occur in non-production environments.
Root cause
Large debug search queries, such as those run through /libs/cq/search/content/querydebug.html, can consume most of the JVM’s memory and CPU. This resource contention leaves insufficient resources for rollout processing, causing rollout jobs to become slow or stuck. The rollout mechanism continues to function but is significantly delayed because of insufficient available resources.
Resolution resolution
Follow the steps below to resolve the issue:
-
Review the use of debug tools such as
/libs/cq/search/content/querydebug.htmlin the AEM Author environment. Avoid running large or unbounded queries, especially during critical rollout periods. -
Restrict access to debug tools. Limit permissions so that only necessary people, such as administrators or developers, can access them. Manage access through user and group permissions in AEM.
-
Monitor JVM resource usage:
- Use available monitoring tools to observe JVM heap and CPU utilization.
- Set up alerts for unusually high memory or CPU consumption.
-
If rollout jobs are stuck and resource contention is confirmed, perform a controlled restart of the affected Author pods to clear the queue and restore normal processing. Coordinate the restart to minimize the impact on ongoing authoring activities.
-
Review user group memberships:
- Check whether people initiating rollouts belong to an unusually large number of groups, because this can increase permission evaluation time and slow down rollouts.
- Compare rollout performance between people with different group memberships to validate the impact.
-
Educate people about the impact of running large debug queries in production environments. Consider implementing query limits or additional monitoring to prevent resource exhaustion from debug tools.
-
Schedule major content launches during periods of low system load when possible.
-
After completing these steps, monitor rollout job progress and system resource usage to confirm that rollout jobs complete as expected.