Resolving intermittent instability in AEM as a Cloud Service after deployment

This article explains how to diagnose and resolve intermittent instability in an Adobe Experience Manager (AEM) as a Cloud Service environment that occurs after a Cloud Manager deployment. It focuses on instability caused by excessive log volume generated by stale workflow instances and provides steps to identify, remediate, and verify the issue.

Description description

Environment

Adobe Experience Manager (AEM) as a Cloud Service (all versions)

Issue/Symptoms

  • The environment becomes intermittently inaccessible or unstable after a recent Cloud Manager deployment.
  • The environment may load successfully at times and then fail without a consistent pattern.
  • Unhealthy pods, pods repeatedly restarting, or inconsistent pod versions may be observed.
  • No specific error codes or stack traces are present.
  • The instability blocks validation and testing activities.

Cause

The instability is typically triggered by excessive log volume generated from stale workflow instances. This can cause the Fluent Bit logging sidecar to exhaust its memory (OOMKilled), leading to repeated restarts and intermittent environment failures.

Resolution resolution

Follow the steps below to resolve the issue:

  1. Verify pod health.

    • Use the Cloud Manager console or Kubernetes monitoring tools to check the health status of all pods in the affected environment.
    • Check for pods in a CrashLoopBackOff or OOMKilled state.
  2. Review recent deployments.

    • Confirm the timing and details of the most recent Cloud Manager deployment that preceded the instability.
    • Review configuration and code changes introduced by the deployment.
  3. Investigate logging components.

    • Check the logs for Fluent Bit sidecar containers for signs of memory exhaustion or repeated restarts.
    • Review the volume of error logs being processed.
  4. Identify and clean up stale workflow instances.

    • Access the AEM Workflow console at /libs/cq/workflow/admin/console/content/instances.html.
    • Filter workflow instances in a RUNNING or SUSPENDED state.
    • Terminate and purge stale workflow instances after obtaining the required approvals.
  5. Monitor environment stability.

    • Monitor pod status and environment accessibility after completing the workflow cleanup.
  6. Address log forwarding issues.

    • Review the log forwarding configuration if log forwarding remains unreliable after workflow cleanup.
    • Check for endpoint limitations.
    • Contact Adobe Support if the issue persists.

Verification: Confirm that the environment is stable, pods are healthy, and validation/testing activities can proceed without interruption.

recommendation-more-help
experience-cloud-kcs-help-kbarticles