Using Config Pipelines config-pipelines

Learn how you can use config pipelines to deploy different configurations AEM as a Cloud Service such as log forwarding settings, purge-related maintenance tasks, and various CDN configurations.

Overview overview

A Cloud Manager config pipeline deploys configurations files (created in YAML format) to a target environment. A number of features in AEM as a Cloud Service can be configured in this way, including log forwarding, purge-related maintenance tasks, and several CDN features.

Config pipelines can be deployed via Cloud Manager to dev, stage, and production environment types in production (non-sandbox) programs. The configuration files can be deployed to Rapid Development Environments (RDEs) using command line tooling.

This following sections of this document give an overview of important information regarding how config pipelines can be used and how configurations for them should be structured. It describes general concepts shared across either all or a subset of the features supported by config pipelines.

Supported Configurations configurations

The following table offers a comprehensive list of such configurations with links to dedicated documentation describing its distinct configuration syntax and other information.

Creating and Managing Config Pipelines creating-and-managing

For information on how to create and configure pipelines, please see the document CI/CD Pipelines.

When creating a config pipeline in Cloud Manager, be sure to select a Targeted Deployment rather than Full Stack Code when configuring the pipeline.

As noted earlier, configuration for RDEs is deployed using command line tooling rather than a pipeline.

Common Syntax common-syntax

Each configuration file begins with properties resembling the following example snippet:

  kind: "LogForwarding"
  version: "1"
  metadata:
    envTypes: ["dev"]

You can use the yq utility to validate locally the YAML formatting of your configuration file (for example, yq cdn.yaml).

Folder Structure folder-structure

A folder named /config or similar should be at the top of the tree, with one more YAML files somewhere in a tree below it.

For example:

/config
  cdn.yaml

or

/config
  /dev
    cdn.yaml

The folder names and filenames below /config are arbitrary. The YAML file, however, must include a valid kind property value.

Typically, configurations are deployed to all environments. If all the property values are identical for each environment, a single YAML file will suffice. However, it is common for property values to differ between environments, for example while testing a lower environment.

The following sections illustrate some strategies for structuring your files.

A Single Config File for All Environments single-file

The file structure will resemble the following:

/config
  cdn.yaml
  logForwarding.yaml

Use this structure when the same configuration is sufficient for all environments and for all types of configuration (CDN, log forwarding, and so on). In this scenario, the envTypes array property would include all environment types.

   kind: "cdn"
   version: "1"
   metadata:
     envTypes: ["dev", "stage", "prod"]

Using secret-type environment variables, it is possible for secret properties to vary per environment, as illustrated by the ${{SPLUNK_TOKEN}} reference

kind: "LogForwarding"
version: "1"
metadata:
  envTypes: ["dev"]
data:
  splunk:
    default:
      enabled: true
      host: "splunk-host.example.com"
      token: "${{SPLUNK_TOKEN}}"
      index: "AEMaaCS"

A Separate File Per Environment Type file-per-env

The file structure will resemble the following:

/config
  cdn-dev.yaml
  cdn-stage.yaml
  cdn-prod.yaml
  logForwarding-dev.yaml
  logForwarding-stage.yaml
  logForwarding-prod.yaml

Use this structure when there may be differences in property values. In the files, one would expect the envTypes array value to correspond with the suffix, for example
cdn-dev.yaml and logForwarding-dev.yaml with a value of ["dev"], cdn-stage.yaml and logForwarding-stage.yaml with a value of ["stage"], and so on.

A Folder Per Environment folder-per-env

In this strategy, there is a separate config folder per environment, with a separate pipeline declared in Cloud Manager for each.

This approach is particularly useful if you have multiple dev environments, where each has unique property values.

The file structure will resemble the following:

/config/dev1
  cdn.yaml
  logForwarding.yaml
/config/dev2
  cdn.yaml
  logForwarding.yaml
/config/prod
  cdn.yaml
  logForwarding.yaml

A variation of this approach is to maintain a separate branch per environment.

Secret Environment Variables secret-env-vars

So that sensitive information need not be stored in source control, configuration files support Cloud Manager environment variables of type secret. For some configurations, including log forwarding, secret environment variables are mandatory for certain properties.

The snippet below is an example of how the secret environment variable ${{SPLUNK_TOKEN}} is used in the configuration.

kind: "LogForwarding"
version: "1"
metadata:
  envTypes: ["dev"]
data:
  splunk:
    default:
      enabled: true
      host: "splunk-host.example.com"
      token: "${{SPLUNK_TOKEN}}"
      index: "AEMaaCS"

Please see the document Cloud Manager Environment Variables for details on how to use environment variables.