AEM as a Cloud Service is a platform for customers to include custom code to create unique experiences for their customer base. With this in mind, the logging service is a critical function to debug and understand code execution on local development, and cloud environments, particularly the AEM as a Cloud Service’s Dev environments.
AEM as a Cloud Service logging settings and log levels are managed in configuration files that are stored as part of the AEM project in Git, and deployed as part of the AEM project via Cloud Manager. Logging in AEM as a Cloud Service can be broken into two logical sets:
Logging at the AEM application level, is handled by three logs:
HTTP requests that are served from Publish tier’s Dispatcher cache or upstream CDN are not reflected in these logs.
AEM as a Cloud Service’s provides access to Java log statements. Developers of applications for AEM should follow general Java logging best practices, logging pertinent statements about the execution of custom code, at the following log levels:
AEM Environment | Log Level | Description | Log Statement Availability |
Development | DEBUG |
Describes what is happening in the application. When DEBUG logging is active, statements providing a clear picture of what activities occur and any key parameters that affect processing are logged. |
|
Stage | WARN |
Describes conditions that have the potential to become errors. When WARN logging is active, only statements indicating conditionals that are approaching sub-optimality are logged. |
|
Production | ERROR |
Describes conditions that indicate a failure, and need to resolve. When ERROR logging is active, only statements indicating failures are logged. ERROR log statements indicate a serious issue that should be resolved as soon as possible. |
|
While Java logging supports several other levels of logging granularity, AEM as a Cloud Service recommends using the three levels described above.
AEM Log levels are set per environment type via OSGi configuration, which in turn are committed to Git, and deployed via Cloud Manager to AEM as a Cloud Service. Because of this, it is best to keep log statements consistent and well known for environment types to ensure the logs available via AEM as Cloud Service are available at the optimal log level without requiring redeployment of application with the updated log level configuration.
Example Log Output
22.06.2020 18:33:30.120 [cm-p12345-e6789-aem-author-86657cbb55-xrnzq] *ERROR* [qtp501076283-1809] io.prometheus.client.dropwizard.DropwizardExports Failed to get value from Gauge
22.06.2020 18:33:30.229 [cm-p12345-e6789-aem-author-86657cbb55-xrnzq] *INFO* [qtp501076283-1805] org.apache.sling.auth.core.impl.SlingAuthenticator getAnonymousResolver: Anonymous access not allowed by configuration - requesting credentials
22.06.2020 18:33:30.370 [cm-p12345-e6789-aem-author-86657cbb55-xrnzq] *INFO* [73.91.59.34 [1592850810364] GET /libs/granite/core/content/login.html HTTP/1.1] org.apache.sling.i18n.impl.JcrResourceBundle Finished loading 0 entries for 'en_US' (basename: <none>) in 4ms
22.06.2020 18:33:30.372 [cm-p12345-e6789-aem-author-86657cbb55-xrnzq] *INFO* [FelixLogListener] org.apache.sling.i18n Service [5126, [java.util.ResourceBundle]] ServiceEvent REGISTERED
22.06.2020 18:33:30.372 [cm-p12345-e6789-aem-author-86657cbb55-xrnzq] *WARN* [73.91.59.34 [1592850810364] GET /libs/granite/core/content/login.html HTTP/1.1] libs.granite.core.components.login.login$jsp j_reason param value 'unknown' cannot be mapped to a valid reason message: ignoring
Log Format
Date and time | 29.04.2020 21:50:13.398 |
AEM as a Cloud Service node ID | [cm-p1234-e5678-aem-author-59555cb5b8-q7l9s] |
Log level | DEBUG |
Thread | qtp2130572036-1472 |
Java class | com.example.approval.workflow.impl.CustomApprovalWorkflow |
Log message | No specified approver, defaulting to [ Creative Approvers user group ] |
AEM Java logs are defined as OSGi configuration, and thus target specific AEM as a Cloud Service environments using run mode folders.
Configure java logging for custom Java packages via OSGi configurations for the Sling LogManager factory. There are two supported configuration properties:
OSGi Configuration property | Description |
---|---|
org.apache.sling.commons.log.names | The Java packages for which to collect log statements. |
org.apache.sling.commons.log.level | The log level at which to log the Java packages, specified by org.apache.sling.commons.log.names |
Changing other LogManager OSGi configuration properties may result in availability issues in AEM as a Cloud Service.
The following are examples of the recommended logging configurations (using the placeholder Java package of com.example
) for the three AEM as a Cloud Service environment types.
/apps/my-app/config/org.apache.sling.commons.log.LogManager.factory.config-example.cfg.json
{
"org.apache.sling.commons.log.names": ["com.example"],
"org.apache.sling.commons.log.level": "debug"
}
/apps/my-app/config.stage/org.apache.sling.commons.log.LogManager.factory.config-example.cfg.json
{
"org.apache.sling.commons.log.names": ["com.example"],
"org.apache.sling.commons.log.level": "warn"
}
/apps/my-app/config.prod/org.apache.sling.commons.log.LogManager.factory.config-example.cfg.json
{
"org.apache.sling.commons.log.names": ["com.example"],
"org.apache.sling.commons.log.level": "error"
}
AEM as a Cloud Service’s HTTP request logging provides insight into the HTTP requests made to AEM and their HTTP responses in time order. This log is helpful to understand the HTTP Requests made to AEM and the order they are processed and responded to.
The key to understanding this log is mapping the HTTP request and response pairs by their IDs, denoted by the numeric value in the brackets. Note that often requests and their corresponding responses have other HTTP requests and responses interjected between them in the log.
Example Log
29/Apr/2020:19:14:21 +0000 [137] -> POST /conf/global/settings/dam/adminui-extension/metadataprofile/ HTTP/1.1 [cm-p1234-e5678-aem-author-59555cb5b8-q7l9s]
...
29/Apr/2020:19:14:22 +0000 [139] -> GET /mnt/overlay/dam/gui/content/processingprofilepage/metadataprofiles/editor.html/conf/global/settings/dam/adminui-extension/metadataprofile/main HTTP/1.1 [cm-p1234-e5678-aem-author-59555cb5b8-q7l9s]
...
29/Apr/2020:19:14:21 +0000 [137] <- 201 text/html 111ms [cm-p1234-e5678-aem-author-59555cb5b8-q7l9s]
...
29/Apr/2020:19:14:22 +0000 [139] <- 200 text/html;charset=utf-8 637ms [cm-p1234-e5678-aem-author-59555cb5b8-q7l9s]
Log Format
Date and time | 29/Apr/2020:19:14:21 +0000 |
Request/Response Pair ID | [137] |
HTTP Method | POST |
URL | /conf/global/settings/dam/adminui-extension/metadataprofile/ |
Protocol | HTTP/1.1 |
AEM as a Cloud Service node ID | [cm-p1234-e5678-aem-author-59555cb5b8-q7l9s] |
The AEM HTTP Request log is not configurable in AEM as a Cloud Service.
AEM as Cloud Service HTTP access logging shows HTTP requests in time order. Each log entry represents the HTTP Request that accesses AEM.
This log is helpful to quickly understand what HTTP requests are being made to AEM, if they succeed by looking at the accompanying HTTP response status code, and how long the HTTP request took to complete. This log can also be helpful to debug a specific user’s activity by filtering log entries by Users.
Example Log Output
cm-p1234-e26813-aem-author-59555cb5b8-8kgr2 - example@adobe.com 30/Apr/2020:17:37:14 +0000 "GET /libs/granite/ui/references/clientlibs/references.lc-5188e85840c529149e6cd29d94e74ad5-lc.min.css HTTP/1.1" 200 1141 "https://author-p10711-e26813.adobeaemcloud.com/mnt/overlay/dam/gui/content/assets/metadataeditor.external.html?item=/content/dam/en/images/example.jpeg&_charset_=utf8" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_4) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/81.0.4044.122 Safari/537.36"
cm-p1234-e26813-aem-author-59555cb5b8-8kgr2 - example@adobe.com 30/Apr/2020:17:37:14 +0000 "GET /libs/dam/gui/coral/components/admin/customthumb/clientlibs.lc-60e4443805c37afa0c74b674b141f1df-lc.min.css HTTP/1.1" 200 809 "https://author-p10711-e26813.adobeaemcloud.com/mnt/overlay/dam/gui/content/assets/metadataeditor.external.html?item=/content/dam/en/images/example.jpeg&_charset_=utf8" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_4) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/81.0.4044.122 Safari/537.36"
cm-p1234-e26813-aem-author-59555cb5b8-8kgr2 - example@adobe.com 30/Apr/2020:17:37:14 +0000 "GET /libs/dam/gui/coral/components/admin/metadataeditor/clientlibs/metadataeditor.lc-4a2226d8232f8b7ab27d24820b9ddd64-lc.min.js HTTP/1.1" 200 7965 "https://author-p10711-e26813.adobeaemcloud.com/mnt/overlay/dam/gui/content/assets/metadataeditor.external.html?item=/content/dam/en/images/example.jpeg&_charset_=utf8" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_4) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/81.0.4044.122 Safari/537.36"
AEM as a Cloud Service Node ID | cm-p1235-e2644-aem-author-59555cb5b8-8kgr2 |
---|---|
IP address of the Client | - |
User | myuser@adobe.com |
Date and time | 30/Apr/2020:17:37:14 +0000 |
HTTP method | GET |
URL | /libs/granite/ui/references/clientlibs/references.lc-5188e85840c529149e6cd29d94e74ad5-lc.min.css |
Protocol | HTTP/1.1 |
HTTP response status | 200 |
Size of the response body in bytes | 1141 |
Referrer | "https://author-p1234-e4444.adobeaemcloud.com/mnt/overlay/dam/gui/content/assets/metadataeditor.external.html?item=/content/dam/wknd/en/adventures/surf-camp-in-costa-rica/adobestock_266405335.jpeg&_charset_=utf8" |
User agent | "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_4) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/81.0.4044.122 Safari/537.36" |
The HTTP Access log is not configurable in AEM as a Cloud Service.
AEM as a Cloud Service provides three logs for the Apache Web Servers and dispatcher layer on the Publish:
Note that these logs are only available for the Publish tier.
This set of logs provides insights into HTTP requests to the AEM as a Cloud Service Publish tier prior to those requests reaching the AEM application. This is important to understand as, ideally, most HTTP requests to the Publish tier servers are served by content that is cached by the Apache HTTPD Web Server and AEM Dispatcher, and never reach the AEM application itself. Thus there are no log statements for these requests in AEM’s Java, Request or Access logs.
The Apache HTTP Web Server access log provides statements for each HTTP request that reaches the Publish tier’s Web server/Dispatcher. Note that requests that are served from an upstream CDN are not reflected in these logs.
See information about the error log format in the official apache documentation.
Example Log Output
cm-p1234-e5678-aem-publish-b86c6b466-qpfvp - - 17/Jul/2020:09:14:41 +0000 "GET /etc.clientlibs/wknd/clientlibs/clientlib-site/resources/images/favicons/favicon-32.png HTTP/1.1" 200 715 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:78.0) Gecko/20100101 Firefox/78.0"
cm-p1234-e5678-aem-publish-b86c6b466-qpfvp - - 17/Jul/2020:09:14:41 +0000 "GET /etc.clientlibs/wknd/clientlibs/clientlib-site/resources/images/favicons/favicon-512.png HTTP/1.1" 200 9631 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:78.0) Gecko/20100101 Firefox/78.0"
cm-p1234-e5678-aem-publish-b86c6b466-qpfvp - - 17/Jul/2020:09:14:42 +0000 "GET /etc.clientlibs/wknd/clientlibs/clientlib-site/resources/images/country-flags/US.svg HTTP/1.1" 200 810 "https://publish-p6902-e30226.adobeaemcloud.com/content/wknd/us/en.html" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:78.0) Gecko/20100101 Firefox/78.0"
Log Format
AEM as a Cloud Servide node ID | cm-p1234-e26813-aem-publish-5c787687c-lqlxr |
IP Address of the client | - |
User | - |
Date and time | 01/May/2020:00:09:46 +0000 |
HTTP Method | GET |
URL | /content/example.html |
Protocol | HTTP/1.1 |
HTTP Response status | 200 |
Size | 310 |
Referer | - |
User agent | "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_4) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/81.0.4044.122 Safari/537.36" |
This log is not configurable in AEM as a Cloud Service.
The Apache HTTP Web Server error log provides statements for each error in the Publish tier’s Web server/Dispatcher.
See information about the error log format in the official apache documentation.
Example Log Output
Fri Jul 17 02:19:48.093820 2020 [mpm_worker:notice] [pid 1:tid 140272153361288] [cm-p1234-e30226-aem-publish-b86c6b466-b9427] AH00292: Apache/2.4.43 (Unix) Communique/4.3.4-20200424 mod_qos/11.63 configured -- resuming normal operations
Fri Jul 17 02:19:48.093874 2020 [core:notice] [pid 1:tid 140272153361288] [cm-p1234-e30226-aem-publish-b86c6b466-b9427] AH00094: Command line: 'httpd -d /etc/httpd -f /etc/httpd/conf/httpd.conf -D FOREGROUND -D ENVIRONMENT_PROD'
Fri Jul 17 02:29:34.517189 2020 [mpm_worker:notice] [pid 1:tid 140293638175624] [cm-p1234-e30226-aem-publish-b496f64bf-5vckp] AH00295: caught SIGTERM, shutting down
Log Format
Date and time | Fri Jul 17 02:16:42.608913 2020 |
Event Level | [mpm_worker:notice] |
Process ID | [pid 1:tid 140715149343624] |
Pod name | [cm-p1234-e56789-aem-publish-b86c6b466-qpfvp] |
Message | AH00094: Command line: 'httpd -d /etc/httpd -f /etc/httpd/conf/httpd.conf -D FOREGROUND -D |
The mod_rewrite log levels are defined by the variable REWRITE_LOG_LEVEL in the file conf.d/variables/global.var
.
It can be set to error, warn, info, debug and trace1 - trace8, with a default value of warn. To debug your RewriteRules, it is recommended to raise the log level to trace2.
See the mod_rewrite module documentation for more information.
To set the log level per environment, use the appropriate conditional branch in the global.var file, as described below:
Define REWRITE_LOG_LEVEL debug
<IfDefine ENVIRONMENT_STAGE>
...
Define REWRITE_LOG_LEVEL warn
...
</IfDefine>
<IfDefine ENVIRONMENT_PROD>
...
Define REWRITE_LOG_LEVEL error
...
</IfDefine>
Example
[17/Jul/2020:23:48:06 +0000] [I] [cm-p12904-e25628-aem-publish-6c5f7c9dbd-mzcvr] "GET /content/wknd/us/en/adventures.html" - 475ms [publishfarm/0] [action miss] "publish-p12904-e25628.adobeaemcloud.com"
[17/Jul/2020:23:48:07 +0000] [I] [cm-p12904-e25628-aem-publish-6c5f7c9dbd-mzcvr] "GET /content/wknd/us/en/adventures/climbing-new-zealand/_jcr_content/root/responsivegrid/carousel/item_1571266094599.coreimg.jpeg/1473680817282/sport-climbing.jpeg" 302 10ms [publishfarm/0] [action none] "publish-p12904-e25628.adobeaemcloud.com"
[17/Jul/2020:23:48:07 +0000] [I] [cm-p12904-e25628-aem-publish-6c5f7c9dbd-mzcvr] "GET /content/wknd/us/en/adventures/ski-touring-mont-blanc/_jcr_content/root/responsivegrid/carousel/item_1571168419252.coreimg.jpeg/1572047288089/adobestock-238230356.jpeg" 302 11ms [publishfarm/0] [action none] "publish-p12904-e25628.adobeaemcloud.com"
Log Format
Date and time | [17/Jul/2020:23:48:16 +0000] |
Pod Name | [cm-p12904-e25628-aem-publish-6c5f7c9dbd-mzcvr] |
Protocol | GET |
URL | /content/experience-fragments/wknd/language-masters/en/contributors/sofia-sjoeberg/master/_jcr_content/root/responsivegrid/image.coreimg.100.500.jpeg/1572236359031/ayo-ogunseinde-237739.jpeg |
Dispatcher response status code | /content/experience-fragments/wknd/language-masters/en/contributors/sofia-sjoeberg/master/_jcr_content/root/responsivegrid/image.coreimg.100.500.jpeg/1572236359031/ayo-ogunseinde-237739.jpeg |
Duration | 1949ms |
Farm | [publishfarm/0] |
Cache status | [action miss] |
Host | "publish-p12904-e25628.adobeaemcloud.com" |
The dispatcher log levels is defined by the variable DISP_LOG_LEVEL in the file conf.d/variables/global.var
.
It can be set to error, warn, info, debug and trace1, with a default value of warn.
While Dispatcher logging supports several other levels of logging granularity, the AEM as a Cloud Service recommends using the levels described below.
To set the log level per environment, use the appropriate conditional branch in the global.var
file, as described below:
Define DISP_LOG_LEVEL debug
<IfDefine ENVIRONMENT_STAGE>
...
Define DISP_LOG_LEVEL warn
...
</IfDefine>
<IfDefine ENVIRONMENT_PROD>
...
Define DISP_LOG_LEVEL error
...
</IfDefine>
For AEM as a Cloud Service environments, debug is the maximal verbosity level. The trace log level is not supported so you should avoid setting it when working in cloud environments.
This feature is being gradually rolled out to customers in early September.
AEM as a Cloud Service provides access to CDN logs, which are useful for use cases including cache hit ratio optimization. The CDN log format cannot be customized and there is no concept of setting it to different modes such as info, warn, or error.
Note that the Splunk forwarding feature does not yet support CDN logs.
Example
{
"timestamp": "2023-05-26T09:20:01+0000",
"ttfb": 19,
"cli_ip": "147.160.230.112",
"cli_country": "CH",
"rid": "974e67f6",
"req_ua": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/14.0.3 Safari/605.1.15",
"host": "example.com",
"url": "/content/hello.png",
"method": "GET",
"res_ctype": "image/png",
"cache": "PASS",
"status": 200,
"res_age": 0,
"pop": "PAR"
}
Log Format
The CDN logs are distinct from the other logs in that it adheres to a json format.
Field Name | Description |
---|---|
timestamp | The time the request started, after TLS termination |
ttfb | Abbreviation for Time To First Byte. The time interval between the request started up to the point before the response body started being streamed. |
cli_ip | The client IP address. |
cli_country | Two-letter ISO 3166-1 alpha-2 country code for the client country. |
rid | The value of the request header used to uniquely identify the request. |
req_ua | The user agent responsible for making a given HTTP request. |
host | The authority that the request is intended for. |
url | The full path, including query parameters. |
method | HTTP method sent by the client, such as “GET” or “POST”. |
res_ctype | The Content-Type used to indicate the original media type of the resource. |
cache | State of the cache. Possible values are HIT, MISS, or PASS |
status | The HTTP status code as an integer value. |
res_age | The amount of time (in seconds) a response has been cached (in all nodes). |
pop | Datacenter of the CDN cache server. |
AEM as a Cloud Service logs for cloud services can be accessed either by downloading through the Cloud Manager interface or by tailing logs at the command line using the using the Adobe I/O command line interface. For more information, see the Cloud Manager logging documentation.
AEM as a Cloud Service SDK provides logs files to support local development.
AEM logs are located in the folder crx-quickstart/logs
, where the following logs can be viewed:
error.log
request.log
access.log
Apache layer logs, including dispatcher, are in the Docker container which holds the Dispatcher. See the Dispatcher documentation for information on how to start the Dispatcher.
To retrieve the logs:
docker ps
to list your containersdocker exec -it <container> /bin/sh
”, where <container>
is the dispatcher container id from the previous step/mnt/var/www/html
/etc/httpd/logs
httpd_access.log
httpd_error.log
dispatcher.log
Logs are also directly printed to the terminal output. Most of the time, these logs should be DEBUG, which can be accomplished by passing in the Debug level as a parameter when running Docker. For example:
DISP_LOG_LEVEL=Debug ./bin/docker_run.sh out docker.for.mac.localhost:4503 8080
In exceptional circumstances, log levels need to be changed to log at a finer granularity in Stage or Production environments.
While this is possible, it requires changes to the log levels in the configuration files in Git from Warn and Error to Debug, and performing a deployment to AEM as a Cloud Service to register these configuration changes with the environments.
Depending on the traffic and the amount of log statement written by Debug, this may result in an adverse performance impact on the environment, therefore, it’s recommended that changes to Stage and Production debug levels are:
Customers who have Splunk accounts may request via customer support ticket that their AEM Cloud Service logs are forwarded to the appropriate index. The logging data is equivalent to what is available through the Cloud Manager log downloads, but customers may find it convenient to use the query features available in the Splunk product.
The network bandwidth associated with logs sent to Splunk are considered part of the customer’s Network I/O usage.
Note that Splunk forwarding does not yet support CDN logs.
In the support request, customers should indicate:
The properties above should be specified for each relevant program/environment type combination. For example, if a customer wanted dev, staging, and production environments, they should provide three sets of information, as indicated below.
Splunk forwarding for sandbox program environments is not supported.
The Splunk forwarding capability is not possible from a dedicated egress IP address.
You should make sure that the initial request includes all dev environment that should be enabled, in addition to the stage/prod environments. Splunk must have an SSL certificate, and be public facing.
If any new dev environments created after the initial request are intended to have Splunk forwarding, but do not have it enabled, an additional request should be made.
Also note that if dev environments have been requested, it is possible that other dev environments not in the request or even sandbox environments will have Splunk forwarding enabled and will share a Splunk index. Customers can use the aem_env_id
field to distinguish between these environments.
Below you will find a sample customer support request:
Program 123, Production Env
splunk-hec-ext.acme.com
Program 123, Stage Env
splunk-hec-ext.acme.com
Program 123, Dev Envs
splunk-hec-ext.acme.com
It may be sufficient for the same Splunk index to be used for each environment, in which case either the aem_env_type
field can be used to differentiate based on the values dev, stage, and prod. If there are multiple dev environments, the aem_env_id
field can also be used. Some organizations may choose a separate index for the production environment’s logs if the associated index limits access to a reduced set of Splunk users.
Here is an example log entry:
aem_env_id: 1242
aem_env_type: dev
aem_program_id: 12314
aem_tier: author
file_path: /var/log/aem/error.log
host: 172.34.200.12
level: INFO
msg: [FelixLogListener] com.adobe.granite.repository Service [5091, [org.apache.jackrabbit.oak.api.jmx.SessionMBean]] ServiceEvent REGISTERED
orig_time: 16.07.2020 08:35:32.346
pod_name: aemloggingall-aem-author-77797d55d4-74zvt
splunk_customer: true