Set up local Dispatcher Tools

Adobe Experience Manager (AEM)'s Dispatcher is a Apache HTTP Web server module that provides a security and performance layer between the CDN and AEM Publish tier. Dispatcher is an integral part of the overall Experience Manager architecture and should be part of local development set up.

The AEM as a Cloud Service SDK includes the recommended Dispatcher Tools version, that facilitates configuring, validating and simulating Dispatcher locally. Dispatcher Tools is comprised of:

• a baseline set of Apache HTTP Web server and Dispatcher configuration files, located in .../dispatcher-sdk-x.x.x/src
• a configuration validator CLI tool, located at .../dispatcher-sdk-x.x.x/bin/validate (Dispatcher SDK 2.0.29+)
• a configuration generation CLI tool, located at .../dispatcher-sdk-x.x.x/bin/validator
• a configuration deployment CLI tool, located at .../dispatcher-sdk-x.x.x/bin/docker_run
• a Docker image that runs Apache HTTP Web server with the Dispatcher module

Note that ~ is used as shorthand for the User’s Directory. In Windows, this is the equivalent of %HOMEPATH%.

Prerequisites

1. Windows users must use Windows 10 Professional
2. Install Experience Manager Publish Quickstart Jar on the local develop machine.
   • Optionally, install the latest AEM reference web site on the local AEM Publish service. This web site is used in this tutorial to visualize a working Dispatcher.
3. Install and start the latest version of Docker (Docker Desktop 2.2.0.5+ / Docker Engine v19.03.9+) on the local development machine.

Download the Dispatcher Tools (as part of the AEM SDK)

The AEM as a Cloud Service SDK, or AEM SDK, contains the Dispatcher Tools used to run Apache HTTP Web server with the Dispatcher module locally for development, as well as the compatible QuickStart Jar.

If the AEM as a Cloud Service SDK has already been downloaded to setup the local AEM runtime, it does not need to be re-downloaded.

1. Log in to experience.adobe.com/#/downloads with your Adobe ID
   • Your Adobe Organization must be provisioned for AEM as a Cloud Service to download the AEM as a Cloud Service SDK
2. Click on the latest AEM SDK result row to download
   • Ensure AEM SDK’s Dispatcher Tools v2.0.29+ is noted in the download description

Extract the Dispatcher Tools from the AEM SDK zip

The version of Dispatcher Tools is different from that of the AEM SDK. Ensure the version of Dispatcher Tools is provided via the AEM SDK version matching the AEM as a Cloud Service version.

1. Unzip the downloaded aem-sdk-xxx.zip file
2. Unpack the Dispatcher Tools into ~/aem-sdk/dispatcher
   • Windows: Unzip aem-sdk-dispatcher-tools-x.x.x-windows.zip into C:\Users\<My User>\aem-sdk\dispatcher (creating missing folders as needed)
   • macOS / Linux: Execute the accompanying shell script aem-sdk-dispatcher-tools-x.x.x-unix.sh to unpack the Dispatcher Tools
     • chmod a+x aem-sdk-dispatcher-tools-x.x.x-unix.sh && ./aem-sdk-dispatcher-tools-x.x.x-unix.sh

Note that all commands issued below assume the current working directory contains the expanding Dispatcher Tools contents.

This video uses macOS for illustrative purposes. The equivalent Windows/Linux commands can be used to achieve similar results

Understand the Dispatcher configuration files

The Dispatcher Tools provides a set of Apache HTTP Web server and Dispatcher configuration files that define behavior for all environments, including local development.

These files are intended to be copied into an Experience Manager Maven project to the dispatcher/src folder, if they do not already exist in the Experience Manager Maven project.

This video uses macOS for illustrative purposes. The equivalent Windows/Linux commands can be used to achieve similar results

A complete description of the configuration files is available in the unpacked Dispatcher Tools as dispatcher-sdk-x.x.x/docs/Config.html.

Validate configurations

Optionally, the Dispatcher and Apache Web server configurations (via httpd -t) can be validated using the validate script (not to be confused with the validator executable).

• Usage:
  • Windows: bin\validate src
  • macOS / Linux: ./bin/validate.sh ./src

Run Dispatcher locally

To run the Dispatcher locally, the Dispatcher configuration files must be generated using the Dispatcher Tools’s validator CLI tool.

• Usage:
  • Windows: bin\validator full -d out src
  • macOS / Linux: ./bin/validator full -d ./out ./src

This command transpiles the configurations into a file-set compatible with the Docker container’s Apache HTTP Web Server.

Once generated, the transpiled configurations are used run Dispatcher locally in the Docker container. It is important to ensure the latest configurations have been validated using validate and output using the validator’s -d option.

• Usage:
  • Windows: bin\docker_run <deployment-folder> <aem-publish-host>:<aem-publish-port> <dispatcher-port>
  • macOS / Linux: ./bin/docker_run.sh <deployment-folder> <aem-publish-host>:<aem-publish-port> <dispatcher-port>

The aem-publish-host can be set to host.docker.internal, a special DNS name Docker provides in the container that resolves to the host machine’s IP. If he host.docker.internal does not resolve, please see the troubleshooting section below.

For example to start the Dispatcher Docker container using the default configuration files provided by the Dispatcher Tools:

1. Generate the deployment-folder, named out by convention, from scratch every time a configuration changes:

   • Windows: del /Q out && bin\validator full -d out src
   • macOS / Linux: rm -rf ./out && ./bin/validator full -d ./out ./src

2. (Re-)start Dispatcher Docker container providing the path to the deployment folder:

   • Windows: bin\docker_run out host.docker.internal:4503 8080
   • macOS / Linux: ./bin/docker_run.sh ./out host.docker.internal:4503 8080

The AEM as a Cloud Service SDK’s Publish Service, running locally on port 4503 will be available through Dispatcher at http://localhost:8080.

To run Dispatcher Tools against an Experience Manager project’s Dispatcher configuration, simply generate the deployment-folder using the project’s dispatcher/src folder.

• Windows:

  $ del -/Q out && bin\validator full -d out <User Directory>/code/my-project/dispatcher/src
  $ bin\docker_run out host.docker.internal:4503 8080
  

• macOS / Linux:

  $ rm -rf ./out && ./bin/validator full -d ./out ~/code/my-project/dispatcher/src
  $ ./bin/docker_run.sh ./out host.docker.internal:4503 8080
  

This video uses macOS for illustrative purposes. The equivalent Windows/Linux commands can be used to achieve similar results

Dispatcher Tools logs

Dispatcher logs are helpful during local development to understand if and why HTTP Requests are blocked. Log level can be set by prefixing the execution of docker_run with environment parameters.

Dispatcher Tools logs are emitted to the standard out when docker_run is run.

Useful parameters for debugging Dispatcher include:

• DISP_LOG_LEVEL=Debug sets Dispatcher module logging to Debug level
  • Default value is: Warn
• REWRITE_LOG_LEVEL=Debug sets Apache HTTP Web server rewrite module logging to Debug level
  • Default value is: Warn
• DISP_RUN_MODE sets the “run mode” of the Dispatcher environment, loading the corresponding run modes Dispatcher configuration files.
  • Defaults to dev
• Valid values: dev, stage, or prod

One or many parameters, can be passed to docker_run

• Windows:

  $ bin\validator full -d out <User Directory>/code/my-project/dispatcher/src
  $ DISP_LOG_LEVEL=Debug REWRITE_LOG_LEVEL=Debug bin\docker_run out host.docker.internal:4503 8080
  

• macOS / Linux:

  $ ./bin/validator full -d out ~/code/my-project/dispatcher/src
  $ DISP_LOG_LEVEL=Debug REWRITE_LOG_LEVEL=Debug ./bin/docker_run.sh out host.docker.internal:4503 8080
  

This video uses macOS for illustrative purposes. The equivalent Windows/Linux commands can be used to achieve similar results

Log file access

Apache web server and AEM Dispatcher logs can be directly accessed in the Docker container:

• Accessing logs in the Docker container
• Copying the Docker logs to the local filesystem

When to update Dispatcher Tools

Dispatcher Tools versions increment less frequently than the Experience Manager, and thus Dispatcher Tools require fewer updates in the local development environment.

The recommended Dispatcher Tools version is that which is bundled with the AEM as a Cloud Service SDK that matches the Experience Manager as a Cloud Service version. The version of AEM as a Cloud Service can be found via Cloud Manager.

• Cloud Manager > Environments, per environment specified by the AEM Release label

Note that Dispatcher Tools version itself will not match the Experience Manager version.

Troubleshooting

docker_run results in ‘Waiting until host.docker.internal is available’ message

host.docker.internal is an hostname provided to the Docker contain that resolves to the host. Per docs.docker.com (macOS, Windows):

From Docker 18.03 onwards our recommendation is to connect to the special DNS name host.docker.internal, which resolves to the internal IP address used by the host

If, when bin/docker_run out host.docker.internal:4503 8080 results in the message Waiting until host.docker.internal is available, then:

1. Ensure the installed version of Docker is 18.03 or greater
2. You may have an local machine set up that is preventing the registration/resolution of the host.docker.internal name. Instead use your local IP.
   • Windows:
     • From the Command Prompt, execute ipconfig, and record the host’s IPv4 Address of the host machine.
     • Then, execute docker_run using this IP address:
       bin\docker_run out <HOST IP>:4503 8080
   • macOS / Linux:
     • From Terminal, execute ifconfig and record the Host inet IP address, usually the en0 device.
     • Then execute docker_run using the host IP address:
       bin/docker_run.sh out <HOST IP>:4503 8080

Example error

$ docker_run out host.docker.internal:4503 8080

Running script /docker_entrypoint.d/10-check-environment.sh
Running script /docker_entrypoint.d/20-create-docroots.sh
Running script /docker_entrypoint.d/30-wait-for-backend.sh
Waiting until host.docker.internal is available

docker_run results in ‘** error: Deployment folder not found’

When running docker_run.cmd, an error displays that reads ** error: Deployment folder not found:. This typically occurs because there are spaces in the path. If possible, remove the spaces in the folder, or move the aem-sdk folder to a path that does not contain spaces.

For example, Windows user folders often are <First name> <Last name>, with a space between. In the example below the folder ...\My User\... contains a space which breaks the local Dispatcher Tools’ docker_run execution. If the spaces are in a Windows user folder, do not attempt to rename this folder as it will break Windows, instead move the aem-sdk folder to a new location your user has permissions to fully modify. Note that instructions that assume the aem-sdk folder is in the user’s home directory will need to be adjusted to the new location.

Example error

$ \Users\My User\aem-sdk\dispatcher>bin\docker_run.cmd out host.internal.docker:4503 8080

'User\aem-sdk\dispatcher\out\*' is not recognized as an internal or external command,
operable program or batch file.
** error: Deployment folder not found: c:\Users\My User\aem-sdk\dispatcher\out

docker_run fails to start on Windows

Running docker_run on Windows can result in the following error, preventing Dispatcher from starting. This is a reported issue with Dispatcher on Windows and will be fixed in a future release.

Example error

$ \Users\MyUser\aem-sdk\dispatcher>bin\docker_run out host.docker.internal:4503 8080

Running script /docker_entrypoint.d/10-check-environment.sh
Running script /docker_entrypoint.d/20-create-docroots.sh
Running script /docker_entrypoint.d/30-wait-for-backend.sh
Waiting until host.docker.internal is available
host.docker.internal resolves to 192.168.65.2
Running script /docker_entrypoint.d/40-generate-allowed-clients.sh
Running script /docker_entrypoint.d/50-check-expiration.sh
Running script /docker_entrypoint.d/60-check-loglevel.sh
Running script /docker_entrypoint.d/70-check-forwarded-host-secret.sh
Starting httpd server
[Sun Feb 09 17:32:22.256020 2020] [dispatcher:warn] [pid 1:tid 140080096570248] Unable to obtain parent directory of /etc/httpd/conf.dispatcher.d/enabled_farms/farms.any: No such file or directory
[Sun Feb 09 17:32:22.256069 2020] [dispatcher:alert] [pid 1:tid 140080096570248] Unable to import config file: /etc/httpd/conf.dispatcher.d/dispatcher.any
[Sun Feb 09 17:32:22.256074 2020] [dispatcher:alert] [pid 1:tid 140080096570248] Dispatcher initialization failed.
AH00016: Configuration Failed

Additional Resources

• Download AEM SDK
• Adobe Cloud Manager
• Download Docker
• Download the AEM Reference Website (WKND)
• Experience Manager Dispatcher Documentation