Log Forwarding log-forwarding

Customers with a license with a logging vendor or who host a logging product can have AEM logs (including Apache/Dispatcher) and CDN logs forwarded to the associated logging destination. AEM as a Cloud Service supports the following logging destinations:

[1] CDN log forwarding to Amazon S3, Dynatrace, and New Relic is in beta. To request access, email aemcs-logforwarding-beta@adobe.com with your program ID. After access is enabled, you also turn on CDN logs for each destination with cdn.enabled: true. See CDN logs for Amazon S3, Dynatrace, and New Relic (beta).

Log forwarding is configured in a self-service manner by declaring a configuration in Git, and can be deployed via Cloud Manager config pipelines to dev, stage, and production environment types. The configuration file can be deployed to Rapid Development Environments (RDEs) using command line tooling.

There is an option for the AEM and Apache/Dispatcher logs to be routed through AEM’s advanced networking infrastructure, such as dedicated egress IP. CDN logs are always sent directly from the CDN and do not use advanced networking. See Transport & Advanced Networking.

How This Article is Organized how-organized

This article is organized in the following way:

Setup setup

Tokens in the configuration (such as ${{SPLUNK_TOKEN}}) represent secrets, which should not be stored in Git. Instead, declare them as Cloud Manager Secret Environment Variables. Make sure to select All as the dropdown value for the Service Applied field, so logs can be forwarded to author, publish, and preview tiers.

It is possible to set different values between CDN logs and AEM logs (including Apache/Dispatcher), by including an additional cdn and/or aem block after the default block, where properties can override those defined in the default block; only the enabled property is required. A possible use case could be to use a different Splunk index for CDN logs, as the example below illustrates.

   kind: "LogForwarding"
   version: "1"
   data:
     splunk:
       default:
         enabled: true
         host: "splunk-host.example.com"
         token: "${{SPLUNK_TOKEN}}"
         index: "AEMaaCS"
       cdn:
         enabled: true
         token: "${{SPLUNK_TOKEN_CDN}}"
         index: "AEMaaCS_CDN"

Another scenario is to disable either forwarding of the CDN logs or AEM logs (including Apache/Dispatcher). For example, to only forward the CDN logs, one can configure the following:

   kind: "LogForwarding"
   version: "1"
   data:
     splunk:
       default:
         enabled: true
         host: "splunk-host.example.com"
         token: "${{SPLUNK_TOKEN}}"
         index: "AEMaaCS"
       aem:
         enabled: false

Transport & Advanced Networking transport-advancednetworking

Some organizations choose to restrict which traffic can be received by the logging destinations, others may require to use ports other than HTTPS (443). If so Advanced Networking will need to be configured before deploying log forwarding configuration.

Use the table below to see what the requirements are for Advanced Networking and Logging configuration based on whether you are using port 443 or not, and whether or not you need your logs to appear from a fixed IP address. The table applies to AEM logs (including Apache/Dispatcher). For CDN logs, see CDN logs and IP allow lists.

For AEM logs (including Apache/Dispatcher), if you have configured Advanced Networking, you can use the aem.advancedNetworking property to forward them from a Dedicated Egress IP address.

The example below shows how to configure logging on a standard HTTPS port with Advanced Networking.

kind: "LogForwarding"
version: "1"
data:
  splunk:
    default:
      enabled: true
      host: "splunk-host.example.com"
      port: 443
      token: "${{SPLUNK_TOKEN}}"
      index: "aemaacs"
    aem:
      advancedNetworking: true

CDN logs and IP allow lists cdn-ip-allowlist

CDN logs are sent directly from the CDN (Fastly), not from AEM as a Cloud Service. This applies to every logging destination:

If the list of shared IP addresses is too large to allow-list, consider sending CDN logs to an https server or (non-Adobe) Azure Blob Store where logic can be written to send the logs out of a known IP to their ultimate destination.

Mutual TLS (mTLS) authentication (beta) mtls

Splunk, Elasticsearch/OpenSearch, and HTTPS destinations support mutual TLS (mTLS). With mTLS, AEM as a Cloud Service presents a client certificate when it connects to your logging destination, so the destination can verify that the logs come from your environment. mTLS applies to AEM logs (including Apache/Dispatcher) and to CDN logs.

mTLS works alongside the destination’s existing authentication. Keep the destination’s existing authentication properties, such as token (Splunk) or user and password (Elasticsearch/OpenSearch). For HTTPS destinations that forward CDN logs, authHeaderName and authHeaderValue are required.

Configuration mtls-configuration

Add an mtls block alongside the destination’s other properties:

The following example forwards AEM and CDN logs to Splunk using mTLS:

kind: "LogForwarding"
version: "1"
data:
  splunk:
    default:
      enabled: true
      host: "splunk-host.example.com"
      token: "${{SPLUNK_TOKEN}}"
      index: "aemaacs"
      mtls:
        cert: "${{SPLUNK_MTLS_CERT}}"
        key: "${{SPLUNK_MTLS_KEY}}"
        ca: "${{SPLUNK_MTLS_CA}}"

The following rules apply:

Supported key formats mtls-key-formats

The first line of the key’s PEM text shows its format:

For an encrypted key, also set keyPasswd to its password.

Encrypted PKCS#8 is recommended for encrypted keys. To convert a key in the older format, use the following command, which asks for the current password and then for the password of the new key:

openssl pkcs8 -topk8 -v2 aes-256-cbc -in client.key -out client-pkcs8.key

Store the content of client-pkcs8.key in the secret referenced by key, and its password in the secret referenced by keyPasswd.

CDN logs mtls-cdn

Like other properties, an mtls block under default applies to both AEM logs and CDN logs. To use a different certificate for CDN logs, set the mtls properties in a cdn block. To use mTLS for only AEM logs or only CDN logs, place the mtls block in the aem or cdn block instead of default.

After you deploy a configuration that enables CDN log forwarding to a destination, it can take several minutes before the first CDN logs arrive. For HTTPS destinations, CDN log delivery also starts only after your server completes the one-time challenge described in HTTPS. The challenge request is sent with the same client certificate, so your server must accept that certificate on the challenge path too.

Certificate rotation and expiry mtls-rotation

To rotate a certificate, update the Cloud Manager secret environment variables that hold the certificate (and the key, if it changed), then re-run the config pipeline. If the log forwarding configuration file itself is unchanged, also make a change to it so that the configuration is applied again. For example, store the new values in new secret environment variables and reference those names in the mtls block.

You may also receive a proactive Actions Center notification about 21 days before a client certificate expires. Track certificate expiry dates yourself rather than relying on this notification.

Logging Destination Configuration logging-destinations

Configurations for the supported logging destinations are listed below, along with any specific considerations.

Amazon S3 amazons3

CDN logs to Amazon S3 are in beta. To request access, email aemcs-logforwarding-beta@adobe.com with your program ID. See CDN logs for Amazon S3, Dynatrace, and New Relic (beta) for how to enable them, and Amazon S3 CDN logs for the file layout and considerations.

kind: "LogForwarding"
version: "1"
data:
  awsS3:
    default:
      enabled: true
      region: "your-bucket-region"
      bucket: "your-bucket-name"
      accessKey: "${{AWS_S3_ACCESS_KEY}}"
      secretAccessKey: "${{AWS_S3_SECRET_ACCESS_KEY}}"

To forward both AEM logs and CDN logs (CDN logs in beta):

kind: "LogForwarding"
version: "1"
data:
  awsS3:
    default:
      enabled: true
      region: "us-east-1"
      bucket: "your-bucket-name"
      accessKey: "${{AWS_S3_ACCESS_KEY}}"
      secretAccessKey: "${{AWS_S3_SECRET_ACCESS_KEY}}"
    cdn:
      enabled: true

To forward only CDN logs (in beta):

kind: "LogForwarding"
version: "1"
data:
  awsS3:
    default:
      enabled: true
      region: "us-east-1"
      bucket: "your-bucket-name"
      accessKey: "${{AWS_S3_ACCESS_KEY}}"
      secretAccessKey: "${{AWS_S3_SECRET_ACCESS_KEY}}"
    aem:
      enabled: false
    cdn:
      enabled: true

In order to use the S3 Log Forwarder, you will need to preconfigure an AWS IAM user with appropriate policy for accessing your S3 bucket. See AWS IAM User Documentation for how to create IAM user credentials.

The IAM policy should allow the user to use s3:PutObject. To forward CDN logs (in beta), also allow s3:AbortMultipartUpload. No other permissions are needed: the user doesn’t need to read or list the bucket. For example:

{
    "Version": "2012-10-17",
    "Statement": [{
        "Effect": "Allow",
        "Action": [
            "s3:PutObject",
            "s3:AbortMultipartUpload"
        ],
        "Resource": "arn:aws:s3:::your-bucket-name/*"
    }]
}

If the bucket only receives CDN logs, you can limit Resource to arn:aws:s3:::your-bucket-name/aemcdn/*.

See AWS Bucket Policy Documentation for more information on how to implement.

Amazon S3 CDN logs (beta) amazons3-cdn

CDN logs to Amazon S3 are in beta. They need access for your program and cdn.enabled: true. See CDN logs for Amazon S3, Dynatrace, and New Relic (beta).

CDN logs are written below the aemcdn/ folder of your bucket. The filename format is YYYY-MM-DDThh:mm:ss.000-<uniqueid>.log.gz. For example:

aemcdn/
   2026-03-04T10:00:00.000-WnFWYN9BpOUs2aOVn4ee.log.gz
   2026-03-04T10:00:00.000-0pX8ub2yQcAmLs7rT1kd.log.gz
   2026-03-04T10:05:00.000-h3JkVv9ZqN2wE5sXyR0a.log.gz

Customers are responsible for processing and managing these files, for example with an S3 lifecycle rule on the aemcdn/ prefix.

Considerations for CDN logs (beta):

Azure Blob Storage azureblob

kind: "LogForwarding"
version: "1"
data:
  azureBlob:
    default:
      enabled: true
      storageAccountName: "example_acc"
      container: "aem_logs"
      sasToken: "${{AZURE_BLOB_SAS_TOKEN}}

A SAS token should be used for authentication. It should be created from the Shared access signature page, rather than on the Shared access token page, and should be configured with these settings:

Here is a screenshot of a sample SAS token configuration:

If logs have stopped being delivered after previously functioning correctly, check whether the SAS token you configured is still valid, as it may have expired.

Azure Blob Storage CDN logs azureblob-cdn

Each of the globally distributed logging servers will produce a new file every few seconds, under the aemcdn folder. Once created, the file will no longer be appended to. The filename format is YYYY-MM-DDThhss.sss-uniqueid.log. E.g., 2024-03-04T10:00:00.000-WnFWYN9BpOUs2aOVn4ee.log.

As an example, at some point in time:

aemcdn/
   2024-03-04T10:00:00.000-abc.log
   2024-03-04T10:00:00.000-def.log

And then 30 seconds later:

aemcdn/
   2024-03-04T10:00:00.000-abc.log
   2024-03-04T10:00:00.000-def.log
   2024-03-04T10:00:30.000-ghi.log
   2024-03-04T10:00:30.000-jkl.log
   2024-03-04T10:00:30.000-mno.log

Each file contains multiple json log entries, each on a separate line. The log entry formats are described under Logging for AEM as a Cloud Service, and each log entry also includes the additional properties mentioned in the Log Entry Formats section below.

Azure Blob Storage AEM logs azureblob-aem

AEM logs (including Apache/Dispatcher) appear below a folder with the following naming convention:

Under each folder, a single file will be created and appended to. Customers are responsible for processing and managing this file so it does not grow too large.

See the log entry formats under Logging for AEM as a Cloud Service. The log entries will also include the additional properties mentioned in the Log Entry Formats section below.

Datadog datadog

kind: "LogForwarding"
version: "1"
data:
  datadog:
    default:
      enabled: true
      host: "http-intake.logs.datadoghq.eu"
      token: "${{DATADOG_API_KEY}}"
      tags:
         tag1: value1
         tag2: value2

Considerations

Elasticsearch and OpenSearch elastic

kind: "LogForwarding"
version: "1"
data:
  elasticsearch:
    default:
      enabled: true
      host: "example.com"
      user: "${{ELASTICSEARCH_USER}}"
      password: "${{ELASTICSEARCH_PASSWORD}}"
      pipeline: "ingest pipeline name"

Considerations

def envType = ctx.aem_env_type != null ? ctx.aem_env_type : 'unknown';
def sourceType = ctx._index;
def date = new SimpleDateFormat('dd_MM_yyyy').format(new Date());
ctx._index = sourceType + "_" + envType + "_" + date;

HTTPS https

kind: "LogForwarding"
version: "1"
data:
  https:
    default:
      enabled: true
      url: "https://example.com/aem_logs/aem"
      authHeaderName: "X-AEMaaCS-Log-Forwarding-Token"
      authHeaderValue: "${{HTTPS_LOG_FORWARDING_TOKEN}}"

Considerations

HTTPS CDN logs https-cdn

Web requests (POSTs) will be sent continuously, with a json payload that is an array of log entries, with the log entry format described under Logging for AEM as a Cloud Service. Additional properties are mentioned in the Log Entry Formats section below.

There is also be a property named sourcetype, which is set to the value aemcdn.

HTTPS AEM logs https-aem

For AEM logs (including apache/dispacher), web requests (POSTs) will be sent continuously, with a json payload that is an array of log entries, with the various log entry formats as described under Logging for AEM as a Cloud Service. Additional properties are mentioned in the Log Entry Formats section below.

There is also be a property named Source-Type, which is set to one of these values:

New Relic Log API newrelic-https

Log Forwarding to New Relic leverages the New Relic HTTPS Log API for ingestion. CDN logs to New Relic are in beta. To request access, email aemcs-logforwarding-beta@adobe.com with your program ID. See CDN logs for Amazon S3, Dynatrace, and New Relic (beta) for how to enable them.

  kind: "LogForwarding"
  version: "1"
  data:
    newRelic:
      default:
        enabled: true
        uri: "https://log-api.newrelic.com/log/v1"
        apiKey: "${{NR_API_KEY}}"

To forward both AEM logs and CDN logs (CDN logs in beta):

  kind: "LogForwarding"
  version: "1"
  data:
    newRelic:
      default:
        enabled: true
        uri: "https://log-api.newrelic.com/log/v1"
        apiKey: "${{NR_API_KEY}}"
      cdn:
        enabled: true

To forward only CDN logs (in beta):

  kind: "LogForwarding"
  version: "1"
  data:
    newRelic:
      default:
        enabled: true
        uri: "https://log-api.newrelic.com/log/v1"
        apiKey: "${{NR_API_KEY}}"
      aem:
        enabled: false
      cdn:
        enabled: true

Considerations

Dynatrace Log API dynatrace-https

Log Forwarding to Dynatrace leverages the Dynatrace HTTPS API for ingestion. CDN logs to Dynatrace are in beta. To request access, email aemcs-logforwarding-beta@adobe.com with your program ID. See CDN logs for Amazon S3, Dynatrace, and New Relic (beta) for how to enable them.

The token requires the Ingest logs (logs.ingest) scope. The same token is used for AEM logs and CDN logs.

  kind: "LogForwarding"
  version: "1"
  data:
    dynatrace:
      default:
        enabled: true
        environmentId: "${{DYNATRACE_ENVID}}"
        token: "${{DYNATRACE_TOKEN}}"

To forward both AEM logs and CDN logs (CDN logs in beta):

  kind: "LogForwarding"
  version: "1"
  data:
    dynatrace:
      default:
        enabled: true
        environmentId: "${{DYNATRACE_ENVID}}"
        token: "${{DYNATRACE_TOKEN}}"
      cdn:
        enabled: true

To forward only CDN logs (in beta):

  kind: "LogForwarding"
  version: "1"
  data:
    dynatrace:
      default:
        enabled: true
        environmentId: "${{DYNATRACE_ENVID}}"
        token: "${{DYNATRACE_TOKEN}}"
      aem:
        enabled: false
      cdn:
        enabled: true

Considerations

Finding CDN logs in Dynatrace (beta) dynatrace-cdn

Dynatrace stores each CDN log entry, as JSON text, in the content of a log record, and uses the entry’s timestamp as the record time. The CDN log properties, such as source, url, and status, are not separate log attributes. The same applies to AEM logs forwarded to Dynatrace. As a result:

Use advanced mode (DQL) instead. To find CDN logs from the last hour:

fetch logs, from: now() - 1h
| filter matchesPhrase(content, "aemcdn")

To parse the CDN log properties and show some of them:

fetch logs, from: now() - 1h
| parse content, "JSON:cdn"
| filter cdn[source] == "aemcdn"
| fields timestamp, cdn[url], cdn[status], cdn[pop], cdn[cache], cdn[aem_tier]

Splunk splunk

kind: "LogForwarding"
version: "1"
data:
  splunk:
    default:
      enabled: true
      host: "splunk-host.example.com"
      token: "${{SPLUNK_TOKEN}}"
      index: "aemaacs"

Considerations

Sumo Logic sumologic

When configuring Sumo Logic for data ingestion you will be presented with an “HTTP Source Address” which provides the host, receiverURI and the private key in a single string. For example:

https://collectors.de.sumologic.com/receiver/v1/http/ZaVnC...

You will need to copy the last section of the URL (without the preceeding /) and add that as a CloudManager Secret Environment Variable as described in the Setup section above, then reference that variable in your configuration. An example is provided below.

kind: "LogForwarding"
version: "1"
data:
  sumoLogic:
    default:
      enabled: true
      collectorURL: "https://collectors.de.sumologic.com/receiver/v1/http"
      privateKey: "${{SUMOLOGIC_PRIVATE_KEY}}"
      index: "aem-logs"

CDN logs for Amazon S3, Dynatrace, and New Relic (beta) cdn-opt-in

CDN log forwarding to Amazon S3, Dynatrace, and New Relic is in beta. Forwarding AEM logs (including Apache/Dispatcher) to these destinations is not affected and doesn’t need a request.

To forward CDN logs to one of these destinations, complete both steps:

Both steps are needed:

The cdn block inherits all other properties from the default block, including the credentials, so enabled: true is the only property you need to add:

kind: "LogForwarding"
version: "1"
data:
  newRelic:
    default:
      enabled: true
      uri: "https://log-api.newrelic.com/log/v1"
      apiKey: "${{NR_API_KEY}}"
    cdn:
      enabled: true

For a program with access, the configuration works as follows:

As with other destinations, properties set in the cdn block override the values from default. To forward only CDN logs, set aem.enabled: false, as in the last row.

To stop forwarding CDN logs, set cdn.enabled: false and run the config pipeline. CDN log forwarding to that destination stops when the configuration is deployed. AEM logs are not affected.

Before you enable CDN logs, estimate the additional volume. Logging vendors that bill by ingested volume, such as Dynatrace and New Relic, charge for CDN logs in the same way as for AEM logs.

To give feedback on the beta, email aemcs-logforwarding-beta@adobe.com.

Log Entry Formats log-formats

See Logging for AEM as a Cloud Service for the format of each respective log type (CDN logs, and AEM logs including Apache/Dispatcher). Some logging destinations add or change properties in CDN log entries, for example the format of timestamp for New Relic. See CDN log entries by logging destination.

Since logs from multiple programs and environments may be forwarded to the same logging destination, in addition to the output described in the logging article, the following properties will be included in each log entry:

As an example, the properties could have the following values:

aem_env_id: 1242
aem_env_type: dev
aem_program_id: 12314
aem_tier: author

Migrating from Legacy Log Forwarding legacy-migration

Before Log Forwarding configuration was achieved through a self-serve model, customers were requested to open support tickets, where Adobe would initiate the integration.

Customers who had been setup in that manner by Adobe are welcome to adapt to the self-serve model at their convenience. There are several reason to make this transition:

When ready to migrate, simply configure the YAML file as described in the preceding sections. Use the Cloud Manager config pipeline to deploy to each of the environments where the configuration should be applied.

It is recommended, but not required, that a configuration is deployed to all environments so they are all under self-serve control. If not, you may forget which environments have been configured by Adobe versus those configured in a self-serve way.