Investigate requests with missing or empty User-Agent headers in AEM as a Cloud Service

This article explains how to investigate and address a high volume of requests with missing or empty User-Agent headers in Adobe Experience Manager (AEM) as a Cloud Service by analyzing CDN logs and reviewing CDN configuration.

Description description

Environment

Adobe Experience Manager as a Cloud Service

Issue/Symptoms

  • CDN logs show a high volume of requests with a missing or empty User-Agent header.
  • You need to identify the request paths, volumes, and sources to determine whether the traffic belongs to an integration, monitoring tool, health check, or unwanted automation.

Cause

A request can have a missing or empty User-Agent when the client does not send the header or an intermediary does not forward it. This alone does not establish whether the request is legitimate or unwanted. Review the relevant CDN logs and configuration to identify the source.

Resolution resolution

Follow the steps below to resolve the issue:

  1. Access the CDN for the affected environment and download the logs for the relevant timeframe.
  2. Analyze the logs by using the AEM CDN Log Analysis Tooling. Refer to the Experience League documentation for guidance on processing and analyzing CDN log data.
  3. Filter the CDN logs for requests in which the User-Agent header is missing or empty.
  4. Identify the URLs, endpoints, and request volumes associated with the filtered requests.
  5. Review the log entries to determine the originating IP addresses, clients, or services generating the requests, and look for patterns that indicate health checks, synthetic monitoring, or integration-related traffic.
  6. Review the CDN configuration and verify that no policy is removing the User-Agent header or preventing it from being forwarded.
  7. Determine whether legitimate monitoring, health-check, or integration traffic is responsible for the missing User-Agent headers.
  8. Do not block all requests that lack a User-Agent header until legitimate traffic sources have been identified and excluded.
  9. After confirming the unwanted traffic sources, implement rate-limiting or blocking controls at the CDN layer or by using AEM CDN traffic-filter rules.
  10. Verify that legitimate traffic continues to function as expected and that any implemented rate-limiting or blocking rules affect only the intended traffic.
recommendation-more-help
experience-cloud-kcs-help-kbarticles