Monitoring and Maintaining Your AEM instance monitoring-and-maintaining-your-aem-instance

After your AEM instances have been deployed certain tasks will be needed to monitor and maintain their operation, performance and integrity.

A key factor here is that to recognize potential issues you need to know how your systems looks and behaves under normal conditions. This is best done by monitoring the system and collecting information over a period of time.

Backups backups

It is good practice to take backups of:

Your company will probably have a backup policy that you will need to follow, additional considerations of what to backup and when include:

Often a full backup is taken at regular intervals (e.g. daily, weekly or monthly), with incremental backups in between (e.g hourly, daily or weekly).

Backing up your software installation backing-up-your-software-installation

After installation, or significant changes in the configuration, take a backup of your software installation.

To do this, you need to back up your entire repository and then:

Backing up your repository backing-up-your-repository

The Backup and Restore section of the CRX documentation covers all issues related to backups of the CRX repository.

For full details of making an online “hot” backup see Creating an Online Backup.

Version Purging version-purging

The Purge Versions tool is intended for purging the versions of a node or a hierarchy of nodes in your repository. Its primary purpose is to help you reduce the size of your repository by removing old versions of your nodes.

This section deals with maintenance operations related to the versioning feature of AEM. The Purge Version tool is intended for purging the versions of a node or a hierarchy of nodes in your repository. Its primary purpose is to help you reduce the size of your repository by removing old versions of your nodes.

Overview overview

The Purge Versions tool is available in the Tools console under Versioning or directly at:

https://<server>:<port>/etc/versioning/purge.html

Start Path An absolute path on which the purge must be done. You can select the Start Path by clicking the repository tree navigator.

Recursive When purging data you can choose between performing the operation on one node or on a whole hierarchy by selecting Recursive. In the last case the given path defines the root node of the hierarchy.

Maximum versions to keep The maximum number of versions to be kept for a node. When these number exceeds this value, the oldest versions are purged.

Maximum version age The maximum age of the version of a node. When the age of a version exceeds this value, it is purged.

Dry Run Because removing versions of your content is definite and can not be reverted without restoring a backup, the Purge Versions tool provides a dry run mode that allows you to preview the purged versions. To launch a dry run of the purge process, click Dry Run.

Purge Launch the purge of the versions on the node defined by the Start Path.

Purging Versions of a Web Site purging-versions-of-a-web-site

To purge versions of a web site, proceed as follows:

Analyzing the Console analyzing-the-console

The Dry Run and Purge processes list all the nodes that have been processed. During the process, a node can have one of the following status:

Moreover the console provides useful information about the versions:

In the next example:

Working with Audit Records and Log Files working-with-audit-records-and-log-files

Auditing records and log files relating to Adobe Experience Manager (AEM) can be found at various locations. The following is provided to give you an overview of what you can find where.

Working with Logs working-with-logs

AEM WCM records detailed logs. After you unpack and start Quickstart, you can find logs in:

Log file rotation log-file-rotation

Log file rotation refers to the process that limits the growth of file by creating new file periodically. In AEM, a log file called error.log will be rotated once a day according to the given rules:

Finding the Log Files finding-the-log-files

Various log files are held on the file server where you installed AEM:

Activating the DEBUG Log Level activating-the-debug-log-level

The default log level Apache Sling Logging Configuration is Information, so debug messages are not logged.

To activate the debug log level for a Logger, set the property org.apache.sling.commons.log.level to debug in the repository. For example, on /libs/sling/config/org.apache.sling.commons.log.LogManager to configure the global Apache Sling Logging.

A line in the debug file usually starts with DEBUG, then provides the log level, the installer action and the log message. For example:

DEBUG 3 WebApp Panel: WebApp successfully deployed

The log levels are as follows:

Create a Custom Log File create-a-custom-log-file

In certain circumstances you may want to create a custom log file with a different log level. You can do this in the repository by:

The Felix Console also provides information about Sling Log Support at ../system/console/slinglog; for example http://localhost:4502/system/console/slinglog.

Finding the Audit Records finding-the-audit-records

Audit records are held to provide a record of who did what and when. Different audit records are generated for both AEM WCM and OSGi events.

AEM WCM Audit records shown when Page Authoring aem-wcm-audit-records-shown-when-page-authoring

AEM WCM Auditing records within the repository aem-wcm-auditing-records-within-the-repository

Within the /var/audit folder, audit records are held according to the resource. You can drill down until you can see the individual records and the information they contain.

These entries hold the same information as shown when editing a page.

OSGi Audit records from the Web Console osgi-audit-records-from-the-web-console

OSGi events also generate audit records which can be seen from the Configuration Status tab -> Log Files tab in the AEM Web Console:

Monitoring Your Replication Agents monitoring-your-replication-agents

You can monitor your replication queues to detect when a queue is either down or blocked - which might in turn indicate a problem with a publishing instance or external system:

To monitor a replication agent:

Again you can develop a solution to detect all replication agents (located under /etc/replication/author or /etc/replication/publish), then check the status of the agent ( enabled, disabled) and the underlying queue ( active, idle, blocked).

Monitoring Performance monitoring-performance

Performance Optimization is an interactive process which receives focus during development. After deployment it is usually reviewed after specific intervals or events.

Methods used while collecting information for optimization can also be used for ongoing monitoring.

The following lists common performance issues which occur, together with proposals on how to spot and counteract them.

Performance issues may stem from a number of causes that have nothing to do with your website, including temporary slowdowns in connection speed, CPU load, and many more.

It may also impact either all your visitors, or only a subset of them.

All this information needs to be obtained, sorted and analyzed before you can either optimize the general performance or solve specific issues.

Tools for Monitoring and Analyzing Performance tools-for-monitoring-and-analyzing-performance

The following gives a short overview of some of the tools available for monitoring and analyzing performance.

Some of these will be dependent on your operating system.

Interpreting the request.log interpreting-the-request-log

This file registers basic information about every request made to AEM. From this valuable conclusions can be extracted.

The request.log offers a built-in way to get a look at how long requests take. For development purposes it is useful to tail -f the request.log and watch for slow response times. To analyze a bigger request.log we recommend the use of rlog.jar which allows you to sort and filter for response times.

We recommend isolating the “slow” pages from the request.log, then individually tuning them for a better performance. This is usually done by including performance metrics per component or using a performance profiling tool such as [yourkit](https://www.yourkit.com/).

Monitoring traffic on your website monitoring-traffic-on-your-website

The request log registers each request made, together with the response made:

09:43:41 [66] -> GET /author/y.html HTTP/1.1
09:43:41 [66] <- 200 text/html 797ms

By totaling all the GET entries within a specific periods (e.g. over various 24 hour periods) you can make statements about the average traffic on your website.

Monitoring response times with the request.log monitoring-response-times-with-the-request-log

A good starting point for performance analysis is the request log:

<cq-installation-dir>/crx-quickstart/logs/request.log

The log looks as follows (the lines are shortened for simplicity):

31/Mar/2009:11:32:57 +0200 [379] -> GET /path/x HTTP/1.1
31/Mar/2009:11:32:57 +0200 [379] <- 200 text/html 33ms
31/Mar/2009:11:33:17 +0200 [380] -> GET /path/y HTTP/1.1
31/Mar/2009:11:33:17 +0200 [380] <- 200 application/json 39ms

This log has one line per request or response:

Using small scripts, you can extract the required information from the log file and assemble the statistics you want. From these, you can see which pages or types of pages are slow, and if the overall performance is satisfactory.

Monitoring search response times with the request.log monitoring-search-response-times-with-the-request-log

Search requests are also registered in the log file:

31/Mar/2009:11:35:34 +0200 [338] -> GET /author/playground/en/tools/search.html?query=dilbert&size=5&dispenc=utf-8 HTTP/1.1
31/Mar/2009:11:35:34 +0200 [338] <- 200 text/html 1562ms

So, as above, you can use scripts to extract the relevant information and build up statistics.

However, once you have determined the response time, you may need to analyze why the request is taking the time it does, and what can be done to improve the response.

Monitoring the number and impact of concurrent users monitoring-the-number-and-impact-of-concurrent-users

Again the request.log can be used to monitor concurrency and the system’s reaction to it.

Tests must be made to determine how many concurrent users the system can handle before a negative impact is seen. Again scripts can be used to extract results from the log file:

31/Mar/2009:11:45:29 +0200 [333] -> GET /author/libs/Personalize/content/statics.close.gif HTTP/1.1
31/Mar/2009:11:45:29 +0200 [334] -> GET /author/libs/Personalize/content/statics.detach.gif HTTP/1.1
31/Mar/2009:11:45:30 +0200 [335] -> GET /author/libs/CFC/content/imgs/logo.rZMNURccynWcTpCxyuBNiTCoiBMmw000.default.gif HTTP/1.1
31/Mar/2009:11:45:32 +0200 [335] <- 304 text/html 0ms
31/Mar/2009:11:45:33 +0200 [334] <- 200 image/gif 31ms
31/Mar/2009:11:45:38 +0200 [333] <- 200 image/gif 31ms
31/Mar/2009:11:45:42 +0200 [336] -> GET /author/libs/CFC/content/imgs/logo.rZMNURccynWcTZRXunQbbQtvuuCMbRRBuWXz0000.default.gif HTTP/1.1
31/Mar/2009:11:45:43 +0200 [337] -> GET /author/titlebar_bg.gif HTTP/1.1
31/Mar/2009:11:45:43 +0200 [336] <- 304 text/html 0ms
31/Mar/2009:11:45:44 +0200 [337] <- 304 text/html 0ms

Using rlog.jar to find requests with long duration times using-rlog-jar-to-find-requests-with-long-duration-times

AEM includes various helper tools located in:
<cq-installation-dir>/crx-quickstart/opt/helpers

One of these, rlog.jar, can be used to quickly sort request.log so that requests are displayed by duration, from longest to shortest time.

The following command shows the possible arguments:

$java -jar rlog.jar
Request Log Analyzer Version 21584 Copyright 2005 Day Management AG
Usage:
  java -jar rlog.jar [options] <filename>
Options:
  -h               Prints this usage.
  -n <maxResults>  Limits output to <maxResults> lines.
  -m <maxRequests> Limits input to <maxRequest> requests.
  -xdev            Exclude POST request to CRXDE.

For example, you can run it specifying request.log file as a parameter and show the 10 first requests that have the longest duration:

$ java -jar ../opt/helpers/rlog.jar -n 10 request.log
*Info * Parsed 464 requests.
*Info * Time for parsing: 22ms
*Info * Time for sorting: 2ms
*Info * Total Memory: 1mb
*Info * Free Memory: 1mb
*Info * Used Memory: 0mb
------------------------------------------------------
     18051ms 31/Mar/2009:11:15:34 +0200 200 GET /content/geometrixx/en/company.html text/ html
      2198ms 31/Mar/2009:11:15:20 +0200 200 GET /libs/cq/widgets.js application/x-javascript
      1981ms 31/Mar/2009:11:15:11 +0200 200 GET /libs/wcm/content/welcome.html text/html
      1973ms 31/Mar/2009:11:15:52 +0200 200 GET /content/campaigns/geometrixx.teasers..html text/html
      1883ms 31/Mar/2009:11:15:20 +0200 200 GET /libs/security/cq-security.js application/x-javascript
      1876ms 31/Mar/2009:11:15:20 +0200 200 GET /libs/tagging/widgets.js application/x-javascript
      1869ms 31/Mar/2009:11:15:20 +0200 200 GET /libs/tagging/widgets/themes/default.js application/x-javascript
      1729ms 30/Mar/2009:16:45:56 +0200 200 GET /libs/wcm/content/welcome.html text/html; charset=utf-8
      1510ms 31/Mar/2009:11:15:34 +0200 200 GET /bin/wcm/contentfinder/asset/view.json/ content/dam?_dc=1238490934657&query=&mimeType=image&_charset_=utf-8 application/json
      1462ms 30/Mar/2009:17:23:08 +0200 200 GET /libs/wcm/content/welcome.html text/html; charset=utf-8

You may need to concatenate the individual request.log files if you need to do this operation on a large data sample.

Apache Bench apache-bench

To minimize the impact of special cases (such as garbage collection, etc), it is recommended to use a tool such as apachebench (see for example, ab for further documentation) to help identify memory leaks and selectively analyze response time.

Apache Bench can be used in the following way:

$ ab -c 5 -k -n 1000 "http://localhost:4503/content/geometrixx/en/company.html"
This is ApacheBench, Version 2.3 <$Revision: 655654 $>
Copyright 1996 Adam Twiss, Zeus Technology Ltd, https://www.zeustech.net/
Licensed to The Apache Software Foundation, https://www.apache.org/

Benchmarking localhost (be patient)
Completed 100 requests
Completed 200 requests
Completed 300 requests
Completed 400 requests
Completed 500 requests
Completed 600 requests
Completed 700 requests
Completed 800 requests
Completed 900 requests
Completed 1000 requests
Finished 1000 requests

Server Software: Day-Servlet-Engine/4.1.52
Server Hostname: localhost
Server Port: 4503

Document Path: /content/geometrixx/en/company.html
Document Length: 24127 bytes

Concurrency Level: 5
Time taken for tests: 69.766 seconds
Complete requests: 1000
Failed requests: 998
(Connect: 0, Receive: 0, Length: 998, Exceptions: 0)
Write errors: 0
Keep-Alive requests: 0
Total transferred: 24160923 bytes
HTML transferred: 24010923 bytes
Requests per second: 14.33 /sec (mean)
Time per request: 348.828 [ms] (mean)
Time per request: 69.766 [ms] (mean, across all concurrent requests)
Transfer rate: 338.20 [Kbytes/sec] received

Connection Times (ms)
min mean[+/-sd] median max
Connect: 0 1 3.9 0 58
Processing: 138 347 568.5 282 8106
Waiting: 137 344 568.1 281 8106
Total: 139 348 568.4 283 8106

Percentage of the requests served within a certain time (ms)
50% 283
66% 323
75% 356
80% 374
90% 439
95% 512
98% 1047
99% 1132
100% 8106 (longest request)

The numbers above are taken from a standard MAcBook Pro laptop (mid 2010) accessing the geometrixx company page, as included in a default AEM installation. The page is very simple, but not optimized for performance.

apachebench also displays the time per request as the mean, across all concurrent requests; see Time per request: 54.595 [ms] (mean, across all concurrent requests). You can change the value of the concurrency parameter -c (number of multiple requests to perform at a time) to see any effects.

Request Counters request-counters

Information about request traffic (number of requests during a specific time period) gives you an indication of the load on your instance. This information can be extracted from request.log, though using counters will automate data collection to let you see:

To automate information collection you can also install a RequestFilter to increment a counter on every request. Multiple counters can be used for different time periods.

The information gathered can be used to indicate:

HTML Comments html-comments

It is recommended that every project includes html comments for server performance. Many good public examples can be found; select a page, open the page source for viewing and scroll to the bottom, code such as the following can be seen:

</body>
 </html>
        <!--
        Page took 58 milliseconds to be rendered by server
         -->

Monitoring Performance using JConsole monitoring-performance-using-jconsole

The tool command jconsole is available with the JDK.

Monitoring Performance using (J)VisualVM monitoring-performance-using-j-visualvm

Since JDK 1.6, the tool command jvisualvm is available. After you have installed JDK 1.6 you can:

You can use this tool to generate thread dumps and memory head dumps. This information is often requested by the technical support team.

Information Collection information-collection

Knowing as much as possible about your installation can help you track down what might have caused a change in performance, and whether these changes are justified. These metrics need to be collected at regular intervals so you can easily see significant changes.

The following information can be useful:

How many authors are working with the system? how-many-authors-are-working-with-the-system

To see the number of authors that have used the system since installation use the command line:

cd <cq-installation-dir>/crx-quickstart/logs
cut -d " " -f 3 access.log | sort -u | wc -l

To see the number of authors working on a given date:

grep "<date>" access.log | cut -d " " -f 3 | sort -u | wc -l

What is the average number of page activations per day? what-is-the-average-number-of-page-activations-per-day

To see the total number of page activations since server installation use a repository query; via CRXDE - Tools - Query:

Then calculate the number of days that have elapsed since installation to calculate the average.

How many pages do you currently maintain on this system? how-many-pages-do-you-currently-maintain-on-this-system

To see the number of pages currently on the server use a repository query; via CRXDE - Tools - Query:

If you use MSM, what is the average number of rollouts per month? if-you-use-msm-what-is-the-average-number-of-rollouts-per-month

To determine the total number of rollouts since installation use a repository query; via CRXDE - Tools - Query:

Calculate the number of months that have elapsed since installation to calculate the average.

What is the average number of Live Copies per month? what-is-the-average-number-of-live-copies-per-month

To determine the total number of Live Copies made since installation use a repository query; via CRXDE - Tools - Query:

Again use the number of months that have elapsed since installation to calculate the average.

If you use AEM Assets, how many assets do you currently maintain in Assets? if-you-use-aem-assets-how-many-assets-do-you-currently-maintain-in-assets

To see how many DAM assets you currently maintain, use a repository query; via CRXDE - Tools - Query:

What is the average size of the assets? what-is-the-average-size-of-the-assets

To determine the total size of the /var/dam folder:

How many templates are currently used? how-many-templates-are-currently-used

To see the number of templates currently on the server use a repository query; via CRXDE - Tools - Query:

How many components are currently used? how-many-components-are-currently-used

To see the number of components currently on the server use a repository query; via CRXDE - Tools - Query:

How many requests per hour do you have on the author system at peak time? how-many-requests-per-hour-do-you-have-on-the-author-system-at-peak-time

To determine the requests per hour you have on the author system at peak time:

How many requests per hour do you have on the publish system at peak time? how-many-requests-per-hour-do-you-have-on-the-publish-system-at-peak-time

Repeat the above procedure on your publish instance.

Analyzing Specific Scenarios analyzing-specific-scenarios

The following is a list of suggestions on what to check if you start experiencing certain performance problems. The list is not (unfortunately) fully comprehensive.

CPU at 100% cpu-at

If the CPU of your system is constantly running at 100% then see:

Out of Memory out-of-memory

Although such errors should be detected during Development and Testing, certain scenarios can slip through.

If your system is running out of memory this can be seen in various ways, including performance degradation and error messages including the subtext:

java.lang.OutOfMemoryError

In these cases check:

Disk I/O disk-i-o

If your system is either running out of diskspace, or you notice disk thrashing starting see:

Regular Performance Degradation regular-performance-degradation

If you see the performance of your instance deteriorating after each reboot (sometimes a week or more later), then the following can be checked:

JVM Tuning jvm-tuning

The Java Virtual Machine (JVM) has significantly improved in respect to tuning (especially since Java 7). Because of this, specifying a reasonable fixed JVM size and using the defaults will often be suitable.

If the default settings are not suitable, then it is important to establish a method to monitor and assess GC performance before attempting to tune the JVM; this can involve monitoring factors including, heap size, algorithm and other aspects.

Some common choices are:

The resulting log can be ingested by a GC visualizer such as:

[https://www.ibm.com/developerworks/library/j-ibmtools2/](https://www.ibm.com/developerworks/library/j-ibmtools2/)

Or JConsole:

This will help you see how much memory is being used, what GC algorithms are being used, how long they take to run, and what effect this has on your application performance. Without this, tuning is just “randomly twiddling knobs”.