Log Forwarding log-forwarding

Customers with a license with a logging vendor or who host a logging product can have AEM logs (including Apache/Dispatcher) and CDN logs forwarded to the associated logging destination. AEM as a Cloud Service supports the following logging destinations:

Log forwarding is configured in a self-service manner by declaring a configuration in Git, and can be deployed via Cloud Manager config pipelines to dev, stage, and production environment types. The configuration file can be deployed to Rapid Development Environments (RDEs) using command line tooling.

There is an option for the AEM and Apache/Dispatcher logs to be routed through AEM’s advanced networking infrastructure, such as dedicated egress IP.

Note that the network bandwidth associated with logs sent to the logging destination are considered part of your organization’s Network I/O usage.

How This Article is Organized how-organized

This article is organized in the following way:

Setup setup

Tokens in the configuration (such as ${{SPLUNK_TOKEN}}) represent secrets, which should not be stored in Git. Instead, declare them as Cloud Manager Secret Environment Variables. Make sure to select All as the dropdown value for the Service Applied field, so logs can be forwarded to author, publish, and preview tiers.

It is possible to set different values between CDN logs and AEM logs (including Apache/Dispatcher), by including an additional cdn and/or aem block after the default block, where properties can override those defined in the default block; only the enabled property is required. A possible use case could be to use a different Splunk index for CDN logs, as the example below illustrates.

   kind: "LogForwarding"
   version: "1"
   data:
     splunk:
       default:
         enabled: true
         host: "splunk-host.example.com"
         token: "${{SPLUNK_TOKEN}}"
         index: "AEMaaCS"
       cdn:
         enabled: true
         token: "${{SPLUNK_TOKEN_CDN}}"
         index: "AEMaaCS_CDN"

Another scenario is to disable either forwarding of the CDN logs or AEM logs (including Apache/Dispatcher). For example, to only forward the CDN logs, one can configure the following:

   kind: "LogForwarding"
   version: "1"
   data:
     splunk:
       default:
         enabled: true
         host: "splunk-host.example.com"
         token: "${{SPLUNK_TOKEN}}"
         index: "AEMaaCS"
       aem:
         enabled: false

Transport & Advanced Networking transport-advancednetworking

Some organizations choose to restrict which traffic can be received by the logging destinations, others may require to use ports other than HTTPS (443). If so Advanced Networking will need to be configured before deploying log forwarding configuration.

Use the table below to see what the requirements are for Advanced Networking and Logging configuration based on whether you are using port 443 or not, and whether or not you need your logs to appear from a fixed IP address.

For AEM logs (including Apache/Dispatcher), if you have configured Advanced Networking, you can use the aem.advancedNetworking property to forward them from a Dedicated Egress IP address.

The example below shows how to configure logging on a standard HTTPS port with Advanced Networking.

kind: "LogForwarding"
version: "1"
data:
  splunk:
    default:
      enabled: true
      host: "splunk-host.example.com"
      port: 443
      token: "${{SPLUNK_TOKEN}}"
      index: "aemaacs"
    aem:
      advancedNetworking: true

For CDN logs, you can allow-list the IP addresses, as described in Fastly documentation - Public IP List. If that list of shared IP addresses is too large, consider sending traffic to an https server or (non-Adobe) Azure Blob Store where logic can be written to send the logs out of a known IP to their ultimate destination.

Mutual TLS (mTLS) authentication mtls

Splunk, Elasticsearch/OpenSearch, and HTTPS destinations support mutual TLS (mTLS). With mTLS, AEM as a Cloud Service presents a client certificate when it connects to your logging destination, so the destination can verify that the logs come from your environment. mTLS applies to AEM logs (including Apache/Dispatcher) and to CDN logs.

mTLS works alongside the destination’s existing authentication. Keep the destination’s existing authentication properties, such as token (Splunk) or user and password (Elasticsearch/OpenSearch). For HTTPS destinations that forward CDN logs, authHeaderName and authHeaderValue are required.

Configuration mtls-configuration

Add an mtls block alongside the destination’s other properties:

The following example forwards AEM and CDN logs to Splunk using mTLS:

kind: "LogForwarding"
version: "1"
data:
  splunk:
    default:
      enabled: true
      host: "splunk-host.example.com"
      token: "${{SPLUNK_TOKEN}}"
      index: "aemaacs"
      mtls:
        cert: "${{SPLUNK_MTLS_CERT}}"
        key: "${{SPLUNK_MTLS_KEY}}"
        ca: "${{SPLUNK_MTLS_CA}}"

The following rules apply:

Supported key formats mtls-key-formats

The first line of the key’s PEM text shows its format:

For an encrypted key, also set keyPasswd to its password.

Encrypted PKCS#8 is recommended for encrypted keys. To convert a key in the older format, use the following command, which asks for the current password and then for the password of the new key:

openssl pkcs8 -topk8 -v2 aes-256-cbc -in client.key -out client-pkcs8.key

Store the content of client-pkcs8.key in the secret referenced by key, and its password in the secret referenced by keyPasswd.

CDN logs mtls-cdn

Like other properties, an mtls block under default applies to both AEM logs and CDN logs. To use a different certificate for CDN logs, set the mtls properties in a cdn block. To use mTLS for only AEM logs or only CDN logs, place the mtls block in the aem or cdn block instead of default.

After you deploy a configuration that enables CDN log forwarding to a destination, it can take several minutes before the first CDN logs arrive. For HTTPS destinations, CDN log delivery also starts only after your server completes the one-time challenge described in HTTPS. The challenge request is sent with the same client certificate, so your server must accept that certificate on the challenge path too.

Certificate rotation and expiry mtls-rotation

To rotate a certificate, update the Cloud Manager secret environment variables that hold the certificate (and the key, if it changed), then re-run the config pipeline. If the log forwarding configuration file itself is unchanged, also make a change to it so that the configuration is applied again. For example, store the new values in new secret environment variables and reference those names in the mtls block.

You may also receive a proactive Actions Center notification about 21 days before a client certificate expires. Track certificate expiry dates yourself rather than relying on this notification.

Logging Destination Configuration logging-destinations

Configurations for the supported logging destinations are listed below, along with any specific considerations.

Amazon S3 amazons3

Log Forwarding to Amazon S3 supports AEM and Dispatcher logs, CDN logs are not yet supported.

kind: "LogForwarding"
version: "1"
data:
  awsS3:
    default:
      enabled: true
      region: "your-bucket-region"
      bucket: "your_bucket_name"
      accessKey: "${{AWS_S3_ACCESS_KEY}}"
      secretAccessKey: "${{AWS_S3_SECRET_ACCESS_KEY}}"

In order to use the S3 Log Forwarder, you will need to preconfigure an AWS IAM user with appropriate policy for accessing your S3 bucket. See AWS IAM User Documentation for how to create IAM user credentials.

The IAM policy should allow the user to use s3:putObject. For example:

 {
    "Version": "2012-10-17",
    "Statement": [{
        "Effect": "Allow",
        "Action": [
            "s3:PutObject"
        ],
        "Resource": "arn:aws:s3:::your_bucket_name/*"
    }]
}

See AWS Bucket Policy Documentation for more information on how to implement.

Azure Blob Storage azureblob

kind: "LogForwarding"
version: "1"
data:
  azureBlob:
    default:
      enabled: true
      storageAccountName: "example_acc"
      container: "aem_logs"
      sasToken: "${{AZURE_BLOB_SAS_TOKEN}}

A SAS token should be used for authentication. It should be created from the Shared access signature page, rather than on the Shared access token page, and should be configured with these settings:

Here is a screenshot of a sample SAS token configuration:

If logs have stopped being delivered after previously functioning correctly, check whether the SAS token you configured is still valid, as it may have expired.

Azure Blob Storage CDN logs azureblob-cdn

Each of the globally distributed logging servers will produce a new file every few seconds, under the aemcdn folder. Once created, the file will no longer be appended to. The filename format is YYYY-MM-DDThhss.sss-uniqueid.log. E.g., 2024-03-04T10:00:00.000-WnFWYN9BpOUs2aOVn4ee.log.

As an example, at some point in time:

aemcdn/
   2024-03-04T10:00:00.000-abc.log
   2024-03-04T10:00:00.000-def.log

And then 30 seconds later:

aemcdn/
   2024-03-04T10:00:00.000-abc.log
   2024-03-04T10:00:00.000-def.log
   2024-03-04T10:00:30.000-ghi.log
   2024-03-04T10:00:30.000-jkl.log
   2024-03-04T10:00:30.000-mno.log

Each file contains multiple json log entries, each on a separate line. The log entry formats are described under Logging for AEM as a Cloud Service, and each log entry also includes the additional properties mentioned in the Log Entry Formats section below.

Azure Blob Storage AEM logs azureblob-aem

AEM logs (including Apache/Dispatcher) appear below a folder with the following naming convention:

Under each folder, a single file will be created and appended to. Customers are responsible for processing and managing this file so it does not grow too large.

See the log entry formats under Logging for AEM as a Cloud Service. The log entries will also include the additional properties mentioned in the Log Entry Formats section below.

Datadog datadog

kind: "LogForwarding"
version: "1"
data:
  datadog:
    default:
      enabled: true
      host: "http-intake.logs.datadoghq.eu"
      token: "${{DATADOG_API_KEY}}"
      tags:
         tag1: value1
         tag2: value2

Considerations

Elasticsearch and OpenSearch elastic

kind: "LogForwarding"
version: "1"
data:
  elasticsearch:
    default:
      enabled: true
      host: "example.com"
      user: "${{ELASTICSEARCH_USER}}"
      password: "${{ELASTICSEARCH_PASSWORD}}"
      pipeline: "ingest pipeline name"

Considerations

def envType = ctx.aem_env_type != null ? ctx.aem_env_type : 'unknown';
def sourceType = ctx._index;
def date = new SimpleDateFormat('dd_MM_yyyy').format(new Date());
ctx._index = sourceType + "_" + envType + "_" + date;

HTTPS https

kind: "LogForwarding"
version: "1"
data:
  https:
    default:
      enabled: true
      url: "https://example.com/aem_logs/aem"
      authHeaderName: "X-AEMaaCS-Log-Forwarding-Token"
      authHeaderValue: "${{HTTPS_LOG_FORWARDING_TOKEN}}"

Considerations

HTTPS CDN logs https-cdn

Web requests (POSTs) will be sent continuously, with a json payload that is an array of log entries, with the log entry format described under Logging for AEM as a Cloud Service. Additional properties are mentioned in the Log Entry Formats section below.

There is also be a property named sourcetype, which is set to the value aemcdn.

HTTPS AEM logs https-aem

For AEM logs (including apache/dispacher), web requests (POSTs) will be sent continuously, with a json payload that is an array of log entries, with the various log entry formats as described under Logging for AEM as a Cloud Service. Additional properties are mentioned in the Log Entry Formats section below.

There is also be a property named Source-Type, which is set to one of these values:

New Relic Log API newrelic-https

Log Forwarding to New Relic leverages the New Relic HTTPS API for ingestion. Currently it only supports logs from AEM and Dispatcher; CDN logs are not yet supported.

  kind: "LogForwarding"
  version: "1"
  data:
    newRelic:
      default:
        enabled: true
        uri: "https://log-api.newrelic.com/log/v1"
        apiKey: "${{NR_API_KEY}}"

Dynatrace Log API dynatrace-https

Log Forwarding to Dynatrace leverages the Dynatrace HTTPS API for ingestion. Currently it only supports logs from AEM and Dispatcher; CDN logs are not yet supported.

The “Ingest Logs” scope attribute is required for the Token.

  kind: "LogForwarding"
  version: "1"
  data:
    dynatrace:
      default:
        enabled: true
        environmentId: "${{DYNATRACE_ENVID}}"
        token: "${{DYNATRACE_TOKEN}}"

Splunk splunk

kind: "LogForwarding"
version: "1"
data:
  splunk:
    default:
      enabled: true
      host: "splunk-host.example.com"
      token: "${{SPLUNK_TOKEN}}"
      index: "aemaacs"

Considerations

Sumo Logic sumologic

When configuring Sumo Logic for data ingestion you will be presented with an “HTTP Source Address” which provides the host, receiverURI and the private key in a single string. For example:

https://collectors.de.sumologic.com/receiver/v1/http/ZaVnC...

You will need to copy the last section of the URL (without the preceeding /) and add that as a CloudManager Secret Environment Variable as described in the Setup section above, then reference that variable in your configuration. An example is provided below.

kind: "LogForwarding"
version: "1"
data:
  sumoLogic:
    default:
      enabled: true
      collectorURL: "https://collectors.de.sumologic.com/receiver/v1/http"
      privateKey: "${{SUMOLOGIC_PRIVATE_KEY}}"
      index: "aem-logs"

Log Entry Formats log-formats

See Logging for AEM as a Cloud Service for the format of each respective log type (CDN logs, and AEM logs including Apache/Dispatcher).

Since logs from multiple programs and environments may be forwarded to the same logging destination, in addition to the output described in the logging article, the following properties will be included in each log entry:

As an example, the properties could have the following values:

aem_env_id: 1242
aem_env_type: dev
aem_program_id: 12314
aem_tier: author

Migrating from Legacy Log Forwarding legacy-migration

Before Log Forwarding configuration was achieved through a self-serve model, customers were requested to open support tickets, where Adobe would initiate the integration.

Customers who had been setup in that manner by Adobe are welcome to adapt to the self-serve model at their convenience. There are several reason to make this transition:

When ready to migrate, simply configure the YAML file as described in the preceding sections. Use the Cloud Manager config pipeline to deploy to each of the environments where the configuration should be applied.

It is recommended, but not required, that a configuration is deployed to all environments so they are all under self-serve control. If not, you may forget which environments have been configured by Adobe versus those configured in a self-serve way.