AEM fails to redirect /robots.txt via Apache rewrite rule

Last update: Wed Jun 04 2025 00:00:00 GMT+0000 (Coordinated Universal Time)

When Apache is configured to rewrite /robots.txt to a DAM path in AEM but the request still returns a 404, it’s likely due to how Dispatcher handles URLs. This article explains how the DispatcherUseProcessedURL setting affects rewrite behavior and how updating it resolves the redirect failure.

Description description

Environment

Adobe Experience Manager (AEM) On-Premise, v6.5

Issue/Symptoms

A rewrite rule is set to redirect /robots.txt to /content/dam/xxxx/robots.txt. The asset exists and is accessible directly.
Dispatcher allow rules permit access to both /robots.txt and /content/dam/xxxx/robots.txt. Despite this, accessing /robots.txt returns a 404 error.

Root cause

The Apache configuration includes the following rewrite rule:

RewriteRule /robots.txt$ /content/dam/xxxx/robots.txt [ PT,L]

This rule correctly rewrites requests for /robots.txt to the DAM path. However, the Dispatcher is set with:

DispatcherUseProcessedURL 0

With this setting, Dispatcher ignores the rewritten URL and instead uses the original request URI. As a result, it doesn’t recognize the rewritten path and returns a 404 error.

According to the Installing Dispatcher document, setting DispatcherUseProcessedURL to 1 ensures Dispatcher uses the processed URL after mod_rewrite, allowing the redirect to work as expected.

Resolution resolution

To resolve the issue:

Open your Apache Dispatcher configuration file, typically named httpd-dispatcher.conf.
Locate the DispatcherUseProcessedURL directive.
Change its value from 0 to 1:
- DispatcherUseProcessedURL 1
Save the file and restart Apache for the changes to take effect.

This change ensures the Dispatcher uses the rewritten URL (/content/dam/xxxx/robots.txt) instead of the original request (/robots.txt), allowing the redirect to work as expected.

The Dispatcher Security Checklist in the AEM Dispatcher Guide

recommendation-more-help