Documentation AEM as a Cloud Service User Guide

UI Testing

Last update: October 23, 2024

CREATED FOR:

Admin
Developer

Custom UI testing is an optional feature that enables you to create and automatically run UI tests for your applications.

Overview

AEM provides an integrated suite of Cloud Manager quality gates to ensure smooth updates to custom applications. In particular, IT test gates already support the creation and automation of custom tests using AEM APIs.

UI tests are packaged in a Docker image to allow a wide choice in language and frameworks (such as Cypress, Selenium, Java and Maven, and JavaScript). Also, a UI tests project can easily be generated by using the AEM Project Archetype.

Adobe encourages the usage of Cypress, as it offers real-time reloading and automatic waiting, which helps save time and improves productivity during testing. Cypress also provides a simple and intuitive syntax, making it easy to learn and use, even for those who are new to testing.

UI tests are executed as part of a specific quality gate for each Cloud Manager pipeline with a Custom UI Testing step in production pipelines or optionally non-production pipelines. Any UI tests including regression and new functionalities enable errors to be detected and reported.

Unlike custom functional tests, which are HTTP tests written in Java, UI tests can be a Docker image with tests written in any language, as long as they follow the conventions defined in the section Building UI Tests.

TIP

Adobe recommends using Cypress for UI testing, following the code provided in the AEM Test Samples repository.

Adobe also provides UI test module examples based on JavaScript with WebdriverIO (see AEM Project Archetype) and Java with WebDriver (see AEM Test Samples repository).

Get Started with UI Tests

This section describes the steps required to set up UI tests for execution in Cloud Manager.

Decide on the test framework that you want to use.
- For Cypress (default), use the sample code from the AEM Test Samples repository or use the sample code that is automatically generated in the ui.tests folder of your Cloud Manager repository.
- For Playwright, use the sample code from the AEM Test Samples repository.
- For Webdriver.IO, use the sample code from the AEM Test Samples repository.
- For Selenium WebDriver, use the sample code from the AEM Test Samples repository.
- For other programming languages, see the section Building UI Tests in this document to set up the test project.
Ensure that UI testing is activated as per the section Customer Opt-In in this document.
Develop your test cases and run the tests locally.
Commit your code into the Cloud Manager repository and execute a Cloud Manager pipeline.

Building UI Tests

A Maven project generates a Docker build context. This Docker build context describes how to create a Docker image containing the UI tests, which Cloud Manager uses to generate a Docker image containing the actual UI tests.

This section describes the steps needed to add a UI tests project to your repository.

TIP

The AEM Project Archetype can generate a UI Tests project for you, which is compliant to the following description, if you do not have special requirements for the programming language.

Generate a Docker Build Context

To generate a Docker build context, you need a Maven module that:

Produces an archive that contains a Dockerfile and every other file necessary to build the Docker image with your tests.
Tags the archive with the ui-test-docker-context classifier.

The simplest way to do this is to configure the Maven Assembly Plugin to create the Docker build context archive and assign the right classifier to it.

You can build UI tests with different technologies and frameworks, but this section assumes that your project is laid out in a way similar to the following.

├── Dockerfile
├── assembly-ui-test-docker-context.xml
├── pom.xml
├── test-module
│   ├── package.json
│   ├── index.js
│   └── wdio.conf.js
└── wait-for-grid.sh

The pom.xml file takes care of the Maven build. Add an execution to the Maven Assembly Plugin similar to the following.

<plugin>
    <groupId>org.apache.maven.plugins</groupId>
    <artifactId>maven-assembly-plugin</artifactId>
    <configuration>
        <descriptors>
            <descriptor>${project.basedir}/assembly-ui-test-docker-context.xml</descriptor>
        </descriptors>
        <tarLongFileMode>gnu</tarLongFileMode>
    </configuration>
    <executions>
        <execution>
            <id>make-assembly</id>
            <phase>package</phase>
            <goals>
                <goal>single</goal>
            </goals>
        </execution>
    </executions>
</plugin>

This execution instructs the Maven Assembly Plugin to create an archive based on the instructions contained in assembly-ui-test-docker-context.xml, called an assembly descriptor in the plugin’s jargon. The assembly descriptor lists all the files that must be part of the archive.

<assembly>
    <id>ui-test-docker-context</id>
    <includeBaseDirectory>false</includeBaseDirectory>
    <formats>
        <format>tar.gz</format>
    </formats>
    <fileSets>
        <fileSet>
            <directory>${basedir}</directory>
            <includes>
                <include>Dockerfile</include>
                <include>wait-for-grid.sh</include>
            </includes>
        </fileSet>
        <fileSet>
            <directory>${basedir}/test-module</directory>
            <excludes>
                <exclude>node/**</exclude>
                <exclude>node_modules/**</exclude>
                <exclude>reports/**</exclude>
            </excludes>
        </fileSet>
    </fileSets>
</assembly>

The assembly descriptor instructs the plugin to create an archive of type .tar.gz and assigns the ui-test-docker-context classifier to it. Moreover, it lists the files that must be included in the archive, including the following:

A Dockerfile, mandatory for building the Docker image
The wait-for-grid.sh script, whose purposes are described below
The actual UI tests, implemented by a Node.js project in the test-module folder

The assembly descriptor also excludes some files that might be generated while running the UI tests locally. This guarantees a smaller archive and faster builds.

The archive containing the Docker build context is automatically picked up by Cloud Manager, which will build the Docker image containing your tests during its deployment pipelines. Eventually, Cloud Manager will run the Docker image to execute the UI tests against your application.

The build should produce either zero or one archive. If it produces zero archives, the test step passes by default. If the build produces more than one archive, which archive is selected is non-deterministic.

Customer Opt-In

For Cloud Manager to build and execute your UI tests, you must opt into this feature by adding a file to your repository.

The file name must be testing.properties.
The file contents must be ui-tests.version=1.
The file must be under the maven submodule for UI tests next to the pom.xml file of the UI tests submodule.
The file must be at the root of the built tar.gz file.

The UI tests build and executions are skipped if this file is not present.

To include a testing.properties file in the build artifact, add an include statement in the assembly-ui-test-docker-context.xml file.

[...]
<includes>
    <include>Dockerfile</include>
    <include>wait-for-grid.sh</include>
    <include>testing.properties</include> <!-- opt-in test module in Cloud Manager -->
</includes>
[...]

NOTE

If your project does not include this line, edit the file to opt into UI testing.

The file may contain a line advising not to edit it. This is due to it being introduced into your project before opt-in UI testing was introduced and clients were not intended to edit the file. This can be safely ignored.

If you are using the samples provided by Adobe:

For the JavaScript-based ui.tests folder generated based from the AEM Project Archetype, you can execute below command to add the required configuration.

echo "ui-tests.version=1" > testing.properties

if ! grep -q "testing.properties" "assembly-ui-test-docker-context.xml"; then
  awk -v line='                <include>testing.properties</include>' '/<include>wait-for-grid.sh<\/include>/ { printf "%s\n%s\n", $0, line; next }; 1' assembly-ui-test-docker-context.xml > assembly-ui-test-docker-context.xml.new && mv assembly-ui-test-docker-context.xml.new assembly-ui-test-docker-context.xml
fi

The Cypress and Java Selenium test samples provided by Adobe already have the opt-in flag set.

Writing UI Tests

This section describes the conventions that the Docker image containing your UI tests must follow. The Docker image is built out of the Docker build context described in the previous section.

Environment Variables

The following environment variables are passed to your Docker image at run time, depending on your framework.

NOTE

These values will be set automatically during pipeline execution - there is no need to set them manually as pipeline variables.

Variable

Examples

Description

Testing Framework

SELENIUM_BASE_URL

http://my-ip:4444

The URL of the Selenium server

Selenium only

SELENIUM_BROWSER

chrome

The browser implementation used by the Selenium Server

Selenium only

AEM_AUTHOR_URL

http://my-ip:4502/context-path

The URL of the AEM author instance

All

AEM_AUTHOR_USERNAME

admin

The user name to log in to the AEM author instance

All

AEM_AUTHOR_PASSWORD

admin

The password to log in to the AEM author instance

All

AEM_PUBLISH_URL

http://my-ip:4503/context-path

The URL of the AEM publish instance

All *

AEM_PUBLISH_USERNAME

admin

The user name to log in to the AEM publish instance

All *

AEM_PUBLISH_PASSWORD

admin

The password to log in to the AEM publish instance

All *

REPORTS_PATH

/usr/src/app/reports

The path where the XML report of the test results must be saved

All

UPLOAD_URL

http://upload-host:9090/upload

The URL where to which the file must be uploaded to make them accessible to the testing framework

All

PROXY_HOST

proxy-host

The hostname of the internal HTTP Proxy to be used by the testing framework

All except Selenium

PROXY_HTTPS_PORT

8071

The proxy server listening port for HTTPS connections (can be empty)

All except Selenium

PROXY_HTTP_PORT

8070

The proxy server listening port for HTTP connections (can be empty)

All except Selenium

PROXY_CA_PATH

/path/to/root_ca.pem

The path to the CA certificate to be used by the testing framework

All except Selenium

PROXY_OBSERVABILITY_PORT

8081

The HTTP healthcheck port of the proxy server

All except Selenium

PROXY_RETRY_ATTEMPTS

12

Suggested number of retry attempts while waiting for proxy server readiness

All except Selenium

PROXY_RETRY_DELAY

5

Suggested delay between retry attempts while waiting for proxy server readiness

All except Selenium

* these values will be empty if there is no publish instance

The Adobe test samples provide helper functions to access the configuration parameters:

Cypress: use the standard function Cypress.env('VARIABLE_NAME')
JavaScript: See the lib/config.js module
Java: See the Config class

Generate Test Reports

The Docker image must generate test reports in the JUnit XML format and save them in the path specified by the environment variable REPORTS_PATH. The JUnit XML format is a widely used format for reporting the results of tests. If the Docker image uses Java and Maven, standard test modules such as Maven Surefire Plugin and Maven Failsafe Plugin can generate such reports out of the box.

If the Docker image is implemented with other programming languages or test runners, check the documentation for the chosen tools for how to generate JUnit XML reports.

NOTE

The result of the UI testing step is evaluated only based on the test reports. Ensure that you generate the report accordingly for your test execution.

Use assertions instead of just logging an error to STDERR or returning a non-zero exit code otherwise your deployment pipeline may proceed normally.

If an HTTP proxy was used during tests execution, the results will include a request.log file.

Prerequisites

The tests in Cloud Manager are run using a technical admin user.

NOTE

For running the functional tests from your local machine, create a user with admin-like permissions to achieve the same behavior.

The containerized infrastructure that is scoped for functional testing is limited by the following boundaries:

Type

Value

Description

CPU

2.0

Amount of CPU-time reserved per test execution.

Memory

1Gi

Amount of memory allocated to the test, value in gibibytes.

Timeout

30m

The duration after which the test is finished.

Recommended Duration

15m

Adobe recommends writing the tests to not take longer than this time.

NOTE

Should you need more resources, create a Customer Care case and describe your use-case; Adobe will review your request and provide appropriate assistance.

Selenium-specific details

NOTE

This section only applies when Selenium is the chosen test infrastructure.

Waiting for Selenium to be Ready

Before the tests start, it’s the responsibility of the Docker image to ensure that the Selenium server is up and running. Waiting for the Selenium service is a two-steps process.

Read the URL of the Selenium service from the SELENIUM_BASE_URL environment variable.
Poll at regular intervals to the status endpoint exposed by the Selenium API.

Once the Selenium’s status endpoint answers with a positive response, the tests can start.

The Adobe UI test samples handle this with the script wait-for-grid.sh, which is executed upon Docker startup and starts the actual test execution only once the grid is ready.

Capture Screenshots and Videos

The Docker image may generate additional test output (for example, screenshots or videos) and save them in the path specified by the environment variable REPORTS_PATH. Any file found below the REPORTS_PATH are included in the test result archive.

The test samples provided by Adobe by default create screenshots for any failed test.

You can use the helper functions to create screenshots through your tests.

JavaScript: takeScreenshot command
Java: Commands

If a test result archive is created during a UI test execution, you can download it from Cloud Manager using the Download Details button under the Custom UI Testing step.

Upload Files

Tests sometimes must upload files to the application being tested. To keep the deployment of Selenium flexible relative to your tests, it is not possible to directly upload an asset directly to Selenium. Instead, uploading a file requires the following steps.

Upload the file at the URL specified by the UPLOAD_URL environment variable.
- The upload must be performed in one POST request with a multipart form.
- The multipart form must have a single file field.
- This is equivalent to curl -X POST ${UPLOAD_URL} -F "data=@file.txt".
- Consult the documentation and libraries of the programming language used in the Docker image to know how to perform such an HTTP request.
- The Adobe test samples provide helper functions for uploading files:
  - JavaScript: See the getFileHandleForUpload command.
  - Java: See the FileHandler class.
If the upload is successful, the request returns a 200 OK response of type text/plain.
- The content of the response is an opaque file handle.
- You can use this handle in place of a file path in an <input> element to test file uploads in your application.

Cypress-specific details

NOTE

This section only applies when Cypress is the chosen test infrastructure.

Set up HTTP proxy

The Docker container’s entrypoint needs to check the value of the PROXY_HOST environment variable.

If this value is empty, no additional steps are required and the tests should be executed without using HTTP proxy.

If it’s not empty, the entrypoint script needs to:

Configure an HTTP proxy connection for running UI tests. This can be achieved by exporting the HTTP_PROXY environment variable that is built using the following values:
- Proxy host, which is provided by PROXY_HOST variable
- Proxy port, which is provided by PROXY_HTTPS_PORT or PROXY_HTTP_PORT variable (the variable with a non-empty value will be used)
Set the CA certificate that will be used when connecting to the HTTP proxy. Its location is provided by PROXY_CA_PATH variable.
- This can be achieved by exporting NODE_EXTRA_CA_CERTS environment variable.
Wait until the HTTP proxy is ready.
- To check the readiness, the environment variables PROXY_HOST, PROXY_OBSERVABILITY_PORT, PROXY_RETRY_ATTEMPTS and PROXY_RETRY_DELAY can be used.
- You can check using a cURL request, making sure to install cURL in your Dockerfile.

An example implementation can be found in the Cypress Sample Test Module’s Entrypoint on GitHub.

Playwright-specific details

NOTE

This section only applies when Playwright is the chosen test infrastructure.

Set up HTTP proxy

NOTE

In the presented examples, we assume Chrome is being used as a project browser.

Similar to Cypress, tests need to use the HTTP proxy if a non-empty PROXY_HOST environment variable is provided.

To do this, the following modifications need to be made.

Dockerfile

Install cURL and libnss3-tools, which provides certutil.

RUN apt -y update \
    && apt -y --no-install-recommends install curl libnss3-tools \
    && rm -rf /var/lib/apt/lists/*

Entrypoint script

Include a bash script that, in case PROXY_HOST environment variable is provided, does the following:

Export proxy-related variables such as HTTP_PROXY and NODE_EXTRA_CA_CERTS
Use certutil to install proxy CA certificate for chromium
Wait until the HTTP proxy is ready (or exit on failure).

Example implementation:

# setup proxy environment variables and CA certificate
if [ -n "${PROXY_HOST:-}" ]; then
  if [ -n "${PROXY_HTTPS_PORT:-}" ]; then
    export HTTP_PROXY="https://${PROXY_HOST}:${PROXY_HTTPS_PORT}"
  elif [ -n "${PROXY_HTTP_PORT:-}" ]; then
    export HTTP_PROXY="http://${PROXY_HOST}:${PROXY_HTTP_PORT}"
  fi
  if [ -n "${PROXY_CA_PATH:-}" ]; then
    echo "installing certificate"
    mkdir -p $HOME/.pki/nssdb
    certutil -d sql:$HOME/.pki/nssdb -A -t "CT,c,c" -n "EaaS Client Proxy Root" -i $PROXY_CA_PATH
    export NODE_EXTRA_CA_CERTS=${PROXY_CA_PATH}
  fi
  if [ -n "${PROXY_OBSERVABILITY_PORT:-}" ] && [ -n "${HTTP_PROXY:-}" ]; then
    echo "waiting for proxy"
    curl --silent  --retry ${PROXY_RETRY_ATTEMPTS:-3} --retry-connrefused --retry-delay ${PROXY_RETRY_DELAY:-10} \
      --proxy ${HTTP_PROXY} --proxy-cacert ${PROXY_CA_PATH:-""} \
      ${PROXY_HOST}:${PROXY_OBSERVABILITY_PORT}
    if [ $? -ne 0 ]; then
      echo "proxy is not ready"
      exit 1
    fi
  fi
fi

Playwright configuration

Modify the playwright configuration (for example in playwright.config.js) to use a proxy in case the HTTP_PROXY environment variable is set.

Example implementation:

const proxyServer = process.env.HTTP_PROXY || ''

// enable proxy if set
if (proxyServer !== '') {
 cfg.use.proxy = {
  server: proxyServer,
 }
}

NOTE

An example implementation can be found in the Playwright Sample Test Module on GitHub.

Running UI Tests Locally

Before activating UI tests in a Cloud Manager pipeline, it’s recommended to run the UI tests locally against the AEM as a Cloud Service SDK or against an actual AEM as a Cloud Service instance.

Cypress Test Sample

Open a shell and navigate to the ui.tests/test-module folder in your repository
Install Cypress and other prerequisites
npm install

Set the environment variables required for test execution

export AEM_AUTHOR_URL=https://author-<program-id>-<environment-id>.adobeaemcloud.com
export AEM_AUTHOR_USERNAME=<user>
export AEM_AUTHOR_PASSWORD=<password>
export AEM_PUBLISH_URL=https://publish-<program-id>-<environment-id>.adobeaemcloud.com
export AEM_PUBLISH_USERNAME=<user>
export AEM_PUBLISH_PASSWORD=<password>
export REPORTS_PATH=target/

Run tests with one of the following commands

npm test              # Using default Cypress browser
npm run test-chrome   # Using Google Chrome browser
npm run test-firefox  # Using Firefox browser

NOTE

The log files are stored in the target/ folder of your repository.

For details, see AEM Test Samples repository.

JavaScript WebdriverIO Test Sample

Open a shell and navigate to the ui.tests folder in your repository

Execute the below command to start the tests using Maven

mvn verify -Pui-tests-local-execution \
 -DAEM_AUTHOR_URL=https://author-<program-id>-<environment-id>.adobeaemcloud.com \
 -DAEM_AUTHOR_USERNAME=<user> \
 -DAEM_AUTHOR_PASSWORD=<password> \
 -DAEM_PUBLISH_URL=https://publish-<program-id>-<environment-id>.adobeaemcloud.com \
 -DAEM_PUBLISH_USERNAME=<user> \
 -DAEM_PUBLISH_PASSWORD=<password>

NOTE

This starts a standalone selenium instance and executes the tests against it.
The log files are stored in the target/reports folder of your repository
You need to ensure that your machine is running the latest Chrome version as the test downloads the latest release of ChromeDriver automatically for testing.

For details, see AEM Test Samples repository.

Playwright Test Sample

Open a shell and navigate to the ui.tests folder in your repository
Execute the below command to build docker image using Maven
mvn clean package -Pui-tests-docker-build

Execute the below command to start the tests using Maven

mvn verify -Pui-tests-docker-execution \
 -DAEM_AUTHOR_URL=https://author-<program-id>-<environment-id>.adobeaemcloud.com \
 -DAEM_AUTHOR_USERNAME=<user> \
 -DAEM_AUTHOR_PASSWORD=<password> \
 -DAEM_PUBLISH_URL=https://publish-<program-id>-<environment-id>.adobeaemcloud.com \
 -DAEM_PUBLISH_USERNAME=<user> \
 -DAEM_PUBLISH_PASSWORD=<password>

NOTE

The log files are stored in the target/ folder of your repository.

For details, see AEM Test Samples repository.

Java Selenium WebDriver Test Sample

Open a shell and navigate to the ui.tests/test-module folder in your repository

Execute the below commands to start the tests using Maven

# Start selenium docker image
# we mount /tmp/shared as a shared volume that will be used between selenium and the test module for uploads
docker run -d -p 4444:4444 -v /tmp/shared:/tmp/shared selenium/standalone-chromium:latest

# Run the tests using the previously started Selenium instance
mvn verify -Pui-tests-local-execution -DSELENIUM_BASE_URL=http://<server>:4444

NOTE

The log files are stored in the target/reports folder of your repository.

For details, see AEM Test Samples repository.

recommendation-more-help