Log Forwarding log-forwarding

Customers with a license with a logging vendor or who host a logging product can have AEM logs (including Apache/Dispatcher) and CDN logs forwarded to the associated logging destination. AEM as a Cloud Service supports the following logging destinations:

Log forwarding is configured in a self-service manner by declaring a configuration in Git, and deploying it via the Cloud Manager Config Pipeline to RDE, dev, stage, and production environment types in production (non-sandbox) programs.

There is an option for the AEM and Apache/Dispatcher logs to be routed through AEM’s advanced networking infrastructure, such as dedicated egress IP.

Note that the network bandwidth associated with logs sent to the logging destination are considered part of your organization’s Network I/O usage.

How This Article is Organized how-organized

This article is organized in the following way:

Setup setup

Tokens in the configuration (such as ${{SPLUNK_TOKEN}}) represent secrets, which should not be stored in Git. Instead, declare them as Cloud Manager Secret Environment Variables. Make sure to select All as the dropdown value for the Service Applied field, so logs can be forwarded to author, publish, and preview tiers.

It is possible to set different values between CDN logs and AEM logs (including Apache/Dispatcher), by including an additional cdn and/or aem block after the default block, where properties can override those defined in the default block; only the enabled property is required. A possible use case could be to use a different Splunk index for CDN logs, as the example below illustrates.

   kind: "LogForwarding"
   version: "1"
   metadata:
     envTypes: ["dev"]
   data:
     splunk:
       default:
         enabled: true
         host: "splunk-host.example.com"
         token: "${{SPLUNK_TOKEN}}"
         index: "AEMaaCS"
       cdn:
         enabled: true
         token: "${{SPLUNK_TOKEN_CDN}}"
         index: "AEMaaCS_CDN"

Another scenario is to disable either forwarding of the CDN logs or AEM logs (including Apache/Dispatcher). For example, to only forward the CDN logs, one can configure the following:

   kind: "LogForwarding"
   version: "1"
   metadata:
     envTypes: ["dev"]
   data:
     splunk:
       default:
         enabled: true
         host: "splunk-host.example.com"
         token: "${{SPLUNK_TOKEN}}"
         index: "AEMaaCS"
       aem:
         enabled: false

Logging Destination Configuration logging-destinations

Configurations for the supported logging destinations are listed below, along with any specific considerations.

Azure Blob Storage azureblob

kind: "LogForwarding"
version: "1"
metadata:
  envTypes: ["dev"]
data:
  azureBlob:
    default:
      enabled: true
      storageAccountName: "example_acc"
      container: "aem_logs"
      sasToken: "${{AZURE_BLOB_SAS_TOKEN}}

A SAS token should be used for authentication. It should be created from the Shared access signature page, rather than on the Shared access token page, and should be configured with these settings:

Here is a screenshot of a sample SAS token configuration:

If logs have stopped being delivered after previously functioning correctly, check whether the SAS token you configured is still valid, as it may have expired.

Azure Blob Storage CDN logs azureblob-cdn

Each of the globally distributed logging servers will produce a new file every few seconds, under the aemcdn folder. Once created, the file will no longer be appended to. The filename format is YYYY-MM-DDThhss.sss-uniqueid.log. E.g., 2024-03-04T10:00:00.000-WnFWYN9BpOUs2aOVn4ee.log.

As an example, at some point in time:

aemcdn/
   2024-03-04T10:00:00.000-abc.log
   2024-03-04T10:00:00.000-def.log

And then 30 seconds later:

aemcdn/
   2024-03-04T10:00:00.000-abc.log
   2024-03-04T10:00:00.000-def.log
   2024-03-04T10:00:30.000-ghi.log
   2024-03-04T10:00:30.000-jkl.log
   2024-03-04T10:00:30.000-mno.log

Each file contains multiple json log entries, each on a separate line. The log entry formats are described under Logging for AEM as a Cloud Service, and each log entry also includes the additional properties mentioned in the Log Entry Formats section below.

Azure Blob Storage AEM logs azureblob-aem

AEM logs (including Apache/Dispatcher) appear below a folder with the following naming convention:

Under each folder, a single file will be created and appended to. Customers are responsible for processing and managing this file so it does not grow too large.

See the log entry formats under Logging for AEM as a Cloud Service. The log entries will also include the additional properties mentioned in the Log Entry Formats section below.

Datadog datadog

kind: "LogForwarding"
version: "1"
metadata:
  envTypes: ["dev"]
data:
  datadog:
    default:
      enabled: true
      host: "http-intake.logs.datadoghq.eu"
      token: "${{DATADOG_API_KEY}}"
      tags:
         tag1: value1
         tag2: value2

Considerations:

Elasticsearch and OpenSearch elastic

kind: "LogForwarding"
version: "1"
metadata:
  envTypes: ["dev"]
data:
  elasticsearch:
    default:
      enabled: true
      host: "example.com"
      user: "${{ELASTICSEARCH_USER}}"
      password: "${{ELASTICSEARCH_PASSWORD}}"
      pipeline: "ingest pipeline name"

Considerations:

def envType = ctx.aem_env_type != null ? ctx.aem_env_type : 'unknown';
def sourceType = ctx._index;
def date = new SimpleDateFormat('dd_MM_yyyy').format(new Date());
ctx._index = sourceType + "_" + envType + "_" + date;

HTTPS https

kind: "LogForwarding"
version: "1"
metadata:
  envTypes: ["dev"]
data:
  https:
    default:
      enabled: true
      url: "https://example.com/aem_logs/aem"
      authHeaderName: "X-AEMaaCS-Log-Forwarding-Token"
      authHeaderValue: "${{HTTPS_LOG_FORWARDING_TOKEN}}"

Considerations:

HTTPS CDN logs https-cdn

Web requests (POSTs) will be sent continuously, with a json payload that is an array of log entries, with the log entry format described under Logging for AEM as a Cloud Service. Additional properties are mentioned in the Log Entry Formats section below.

There is also be a property named sourcetype, which is set to the value aemcdn.

HTTPS AEM logs https-aem

For AEM logs (including apache/dispacher), web requests (POSTs) will be sent continuously, with a json payload that is an array of log entries, with the various log entry formats as described under Logging for AEM as a Cloud Service. Additional properties are mentioned in the Log Entry Formats section below.

There is also be a property named Source-Type, which is set to one of these values:

Splunk splunk

kind: "LogForwarding"
version: "1"
metadata:
  envTypes: ["dev"]
data:
  splunk:
    default:
      enabled: true
      host: "splunk-host.example.com"
      token: "${{SPLUNK_TOKEN}}"
      index: "aemaacs"

Considerations:

Log Entry Formats log-formats

See Logging for AEM as a Cloud Service for the format of each respective log type (CDN logs, and AEM logs including Apache/Dispatcher).

Since logs from multiple programs and environments may be forwarded to the same logging destination, in addition to the output described in the logging article, the following properties will be included in each log entry:

As an example, the properties could have the following values:

aem_env_id: 1242
aem_env_type: dev
aem_program_id: 12314
aem_tier: author

Advanced Networking advanced-networking

Some organizations choose to restrict which traffic can be received by the logging destinations.

For the CDN log, you can allow-list the IP addresses, as described in fastly documentation - Public IP List. If that list of shared IP addresses is too large, consider sending traffic to an https server or (non-Adobe) Azure Blob Store where logic can be written to send the logs out of a known IP to their ultimate destination.

For AEM logs (including Apache/Dispatcher), if you have configured advanced networking, you can use the advancedNetworking property to forward them from a dedicated egress IP address or over a VPN.

kind: "LogForwarding"
version: "1"
metadata:
  envTypes: ["dev"]
data:
  splunk:
    default:
      enabled: true
      host: "splunk-host.example.com"
      port: 443
      token: "${{SPLUNK_TOKEN}}"
      index: "aemaacs"
    aem:
      advancedNetworking: true

Migrating from Legacy Log Forwarding legacy-migration

Before Log Forwarding configuration was achieved through a self-serve model, customers were requested to open support tickets, where Adobe would initiate the integration.

Customers who had been setup in that manner by Adobe are welcome to adapt to the self-serve model at their convenience. There are several reason to make this transition:

When ready to migrate, simply configure the YAML file as described in the preceding sections. Use the Cloud Manager config pipeline to deploy to each of the environments where the configuration should be applied.

It is recommended, but not required, that a configuration is deployed to all environments so they are all under self-serve control. If not, you may forget which environments have been configured by Adobe versus those configured in a self-serve way.