Resolving AEM production outages caused by thread exhaustion from large asset requests
This article explains how to resolve a production outage in Adobe Experience Manager (AEM) caused by excessive concurrent requests for a large video asset, which can exhaust request-processing threads and disrupt service.
Description description
Environment
Adobe Experience Manager (all versions)
Issue/Symptoms
In AEM production environments, serving large video assets to many concurrent users can lead to thread exhaustion if the asset is not cached at the Dispatcher layer. If a cache flush removes the large asset from cache while abnormal or high-volume traffic persists, all requests for the asset are forwarded to AEM Publish. This can quickly consume all available request-processing threads, resulting in service degradation or a complete outage.
- All users are unable to access the production environment (author, publish, or domain).
- Service degradation occurs after a Dispatcher cache flush.
- High thread consumption occurs on AEM Publish.
Cause
A cache flush removed a large video asset from the Dispatcher cache while abnormal or high-volume traffic persisted. This caused all requests for the asset to be forwarded to AEM Publish, exhausting request-processing threads and resulting in a production outage.
Resolution resolution
Follow these steps to resolve the issue:
- Retrieve the affected large video asset from AEM Author and manually cache it at the Apache/Dispatcher layer.
- Apply an immutable flag to the cached asset in Dispatcher to prevent it from being removed by subsequent cache flush operations. Refer to your Dispatcher configuration for details on setting immutable cache rules.
- Monitor the environment to confirm that requests for the video asset are served by Apache/Dispatcher and aren’t forwarded to AEM Publish. Check Dispatcher logs for cache hits and misses related to the asset.
- Investigate the source and nature of the abnormal request pattern that triggered the issue. Review recent traffic patterns, application logs, and any recent changes to identify the cause of the high request volume.
- Validate your application’s video delivery, HTTP range-request, and caching behavior to ensure optimal performance and prevent recurrence.
- After confirming that the asset is served from cache and thread usage has normalized, verify that the production environment is stable and accessible to all users.