Internal crawler traffic and content request usage calculations
This article describes how Adobe Experience Manager (AEM) as a Cloud Service identifies and handles internal crawler traffic in usage calculations and clarifies the exclusion mechanisms in place.
Description description
Environment
Adobe Experience Manager as a Cloud Service
Issue/Symptoms
Organizations using search solutions may run internal crawlers that index AEM content. These crawlers can generate a high volume of requests, which may appear in traffic logs and raise concerns about their impact on billable usage calculations.
Cause
Internal crawlers typically use distinct User-Agent strings and known IP ranges. There may be concerns that such system-to-system traffic is counted toward billable AEMaaCS content requests, rather than being excluded as non-user traffic.
Resolution resolution
To fix concerns about internal crawler traffic affecting content request usage calculations, follow these steps:
- Ensure your internal crawler uses a distinct
User-Agentstring and, if possible, operates from known IP ranges. - Confirm that the crawler’s request paths are consistent with expected crawler activity (such as sitemap-driven URLs).
- Review your AEMaaCS usage reports and traffic logs to identify crawler traffic patterns.
- Adobe’s usage calculation for Content Requests is based on publish-tier traffic. Certain known bots and system traffic are excluded from usage calculations based on recognized signatures (
User-Agentor IP). - Internal or customer-specific crawlers are excluded from usage calculations only if they match an established exclusion pattern. If the crawler uses a recognized signature, its traffic is not counted toward billable usage.
- If you observe unexpected usage attributed to your crawler, gather relevant log details (such as
User-Agent, IP range, and request paths) and contact Adobe Support for further review.
Note: After confirming the crawler’s signature, review your usage reports to verify that its traffic is excluded from billable content request counts.