DocumentationExperience Cloud KCS

Custom headers not passing through Dispatcher or affecting caching behavior

Last update: July 6, 2026

Custom response headers such as X-Robots-Tag, ETag, and custom cache headers disappear when a page is served through the Dispatcher, or they break CDN caching when a legacy Vary: User-Agent directive is present. The Dispatcher forwards only response headers explicitly listed in its /cache/headers and /headers configuration, and Apache overrides some headers (such as ETag) unless told to defer to AEM. Allowing the header in the farm configuration, ensuring Apache forwards it unmodified, removing the legacy Vary: User-Agent directive, and clearing the cache restores correct header passthrough and caching.

Description description

Environments:

  • Adobe Experience Manager as a Cloud Service (AEMaaCS)
  • AEM Managed Services (AMS)
  • Adobe Experience Manager On-Premises Software
  • AEM Dispatcher (Apache) and the CDN layer

Issue/Symptoms:

  • A header is visible on AEM Publish with curl -I but missing on the Dispatcher.
  • The CDN doesn’t cache pages because of Vary: User-Agent.
  • Custom headers appear only when a query parameter (?a=b) is added, because the Dispatcher bypasses the cache.
  • The ETag is lost or modified, such as Apache appending -gzip.
  • The Dispatcher returns outdated content because cached headers aren’t refreshed.

Root cause:

The Dispatcher forwards only response headers explicitly listed in /cache/headers and /headers. When a header isn’t present, the Dispatcher strips it or uses cached header metadata. Apache overrides headers (such as ETag) unless configured to defer to AEM. Some legacy vhost configurations include Header append Vary User-Agent, which breaks CDN caching because of excessive vary cardinality. Query-string requests bypass the cache, which makes missing headers appear only on cached paths.

How to confirm

  1. Determine whether the header is missing before or after the Dispatcher. Run curl -I against both the Publish and Dispatcher URLs and compare the header sets. If the header appears on Publish but is missing or modified on the Dispatcher, the Dispatcher or Apache is stripping it. If it’s already missing on Publish, fix the AEM code first.
  2. Check whether the Dispatcher cache is involved. Re-test with a query parameter (curl -I against a URL with ?a=b). If the header appears only with the query parameter, the resource is being served from cache without stored header metadata.

Resolution resolution

Follow these steps to resolve the issue:

  1. Allow the header for caching in the farm file. In the publish farm .any file, add the header name to the /cache/headers block (For example: X-Robots-Tag, ETag, or Edge-Cache-Tag). After a restart and cache clear, run curl -I and confirm the header appears.

  2. Ensure Apache forwards the header without overwriting it. For AMS and on-premises, edit the vhost:

    • For ETag, add FileETag None and Header unset ETag.
    • For a custom header, set SetEnvIfNoCase X-Robots-Tag "(.+)$" X_ROBOTS_TAG=$1 and Header always set X-Robots-Tag "%{X-Robots-Tag}e" env=X_ROBOTS_TAG.

Run apachectl -t, restart the service, and confirm curl -I shows the header.
3. Allow the header for forwarding (not caching) in the farm file. Add the header to the /headers block so the Dispatcher forwards it to the client even when the response isn’t cached, then confirm with curl -I.
4. Remove the legacy Vary: User-Agent directive. Search the vhost directories with grep -R "Vary User-Agent" . and comment out any instances found, so the CDN resumes caching responses.
5. Clear the cache after the configuration changes. On AEMaaCS, use Cloud Manager → Environment → Clear Cache. On AMS or on-premises, remove the cached files at <dispatcher_cache_path> or trigger an activation flush, then restart Apache. Confirm curl -I shows the correct header.
6. Validate the domain-to-farm mapping. Confirm the vhost includes the requested domain so the Dispatcher uses the correct publish farm with the header rules. Check the Dispatcher logs for No farm matches host; if it appears, add the domain to the vhost ServerName or ServerAlias and retry.

Validation

  1. Run curl -I against the Dispatcher URL and confirm the header is present and matches Publish.
  2. Confirm the cached metadata file (<path>.headers) is created in the Dispatcher cache directory.
  3. Confirm no unintended headers (such as Vary: User-Agent) remain in the vhosts, and check the CDN cache hit ratio before and after the change.
recommendation-more-help
experience-cloud-kcs-help-kbarticles