Offloading distributes processing tasks amoung Experience Manager instances in a topology. With offloading, you can use specific Experience Manager instances for performing specific types of processing. Specialized processing enables you to maximize the usage of available server resources.
Offloading is based on the Apache Sling Discovery and Sling JobManager features. To use offloading, you add Experience Manager clusters to a topology and identify the job topics that the cluster process. Clusters are comprised of one or more Experience Manager instances, so that a single instance is considered to be a cluster.
For information about adding instances to a topology, see Administering Topologies.
The Sling JobManager and JobConsumer enable the creation of jobs that are processed in a topology:
When JobManager creates a job, the Offloading framework selects an Experience Manager cluster in the topology to execute the job:
See Configuring Topic Consumption for information about refining job distribution.
When the Offloading framework selects a cluster to execute a job, and the cluseter is comprised of multiple instances, Sling Distribution determines which instance in the cluster executes the job.
The Offloading framework supports job payloads that associate jobs with resources in the repository. Job payloads are useful when jobs are created for processing resources and the job is offloaded to another computer.
Upon creation of a job, the payload is only guaranteed to be located on the instance that creates the job. When offloading the job, replication agents ensure that the payload is created on the instance that eventually consumes the job. When job execution is complete, reverse replication causes the payload to be copied back to the instance that created the job.
Topologies are loosely-coupled Experience Manager clusters that are participating in offloading. A cluster consists of one or more Experience Manager server instances (a single instance is considered as a cluster).
Each Experience Manager instance runs the following Offloading-related services:
The Discovery Service of all members of the topology point to the Topology Connector on one of the members. In the sections that follow, this member is referred to as the root member.
Each cluster in the topology contains an instance that is recognized as the leader. The cluster leader interacts with the topology on behalf of the other members of the cluster. When the leader leaves the cluster, a new leader for the cluster is automatically chosen.
Use Topology Browser to explore the state of the topology in which the Experience Manager instance is participating. Topology Browser shows the clusters and instances of the topology.
For each cluster, you see a list of cluster members that indicates the order in which each member joined the cluseter, and which member is the Leader. The Current property indicates the instance that you are currently administering.
For each instance in the cluster, you can see several topology-related properties:
Using the Touch UI, click the Tools tab. (http://localhost:4502/tools.html)
In the Granite Operations area, click Offloading Browser.
In the navigation panel, click Topology Browser.
The clusters that are participating in the topology appear.
Click a cluster to see a list of the instances in the cluster and their ID, Current status, and Leader status.
Click an instance ID to see more detailed properties.
You can also use the Web Console to view topology information. The console provides further information about the topology clusters:
Use the following procedure to open the Topology Management page of the Web Console:
Open the Web Console in your browser. (http://localhost:4502/system/console)
Click Main > Topology Management.
The Apache Sling Resource-Based Discovery Service runs on each instance to control how Experience Manager instances interact with a topology.
The Discovery Service sends periodic POST requests (heartbeats) to Topology Connector services to establish and maintain connections with the topology. The Topology Connector service maintains a allow list of IP addresses or host names that are allowed to join the topology:
Use the Web Console or a sling:OsgiConfig node to configure the following properties of the org.apache.sling.discovery.impt.Config service:
Property Name | OSGi Name | Description | Default Value |
---|---|---|---|
Heartbeat timeout (seconds) | heartbeatTimeout | The amount of time in seconds to wait for a heartbeat response before the targeted instance is considered unavailable. | 20 |
Heartbeat interval (seconds) | heartbeatInterval | The amount of time in seconds between heartbeats. | 15 |
Minimal Event Delay (seconds) | minEventDelay | When a change occurs to the topology, the amount of time to delay the change of state from TOPOLOGY_CHANGING to TOPOLOGY_CHANGED. Each change that occurs when the state is TOPOLOGY_CHANGING increases the delay by this amount of time. This delay prevents listeners from being flooded with events. To use no delay, specify 0 or a negative number. |
3 |
Topology Connector URLs | topologyConnectorUrls | The URLs of the Topology Connector services to send heartbeat messages. | http://localhost:4502/libs/sling/topology/connector |
Topology Connector allow list | topologyConnectorWhitelist | The list of IP addresses or host names that the local Topology Connector service allows in the topology. | localhost 127.0.0.1 |
Repository Descriptor Name | leaderElectionRepositoryDescriptor | <no value> |
Use the following procedure to connect a CQ instance to the root member of a topology. The procedure points the instance to the Topology Connector URL of the root topology member. Perform this procedure on all members of the topology.
Perform the following procedure on the root member of the topology. The procedure adds the names of the other topology members to its Discovery Service allow list.
Use Offloading Browser to configure topic consumption for the Experience Manager instances in the topology. For each instance, you can specify the topics that it consumes. For example, to configure your topology so that only one instance consumes topics of a specific type, disable the topic on all instances except for one.
Jobs are distributed amoung instances that have the associated topic enabled using round-robin logic.
Using the Touch UI, click the Tools tab. (http://localhost:4502/tools.html)
In the Granite Operations area, click Offloading Browser.
In the navigation panel, click Offloading Browser.
The offloading topics and the server instances that that can consume the topics appear.
To disable the consumption of a topic for an instance, below the topc name click Disable beside the instance.
To configure all topic consumption for an instance, click the instance identifier below any topic.
Click one of the following buttons beside a topic to configure the consumption behavior for the instance, and then click Save:
Note: When you select Exclusive for a topic, all of the other topics are automatically set to Disabled.
Several JobConsumer implementations are installed with Experience Manager. The topics for which these JobConsumers are registered appear in Offloading Browser. Additional topics that appear are those that custom JobConsumers have registered. The following table describes the default JobConsumers.
Job topic | Service PID | Description |
---|---|---|
/ | org.apache.sling.event.impl.jobs.deprecated.EventAdminBridge | Installed with Apache Sling. Processes jobs that the OSGi event admin generates, for backward compatibility. |
com/day/cq/replication/job/* | com.day.cq.replication.impl.AgentManagerImpl | A replication agent that replicates job payloads. |
com/adobe/granite/workflow/offloading | com.adobe.granite.workflow.core.offloading.WorkflowOffloadingJobConsumer | Processes jobs that the DAM Update Asset Offloader workflow generates. |
The Apache Sling Job Consumer Manager service provides topic allow list and block list properties. Configure these properties to enable or disable the processing of specific topics on an Experience Manager instance.
Note: If the instance belongs to a topology, you can also use Offloading Browser on any computer in the topology to enable or disable topics.
The logic that creates the list of enabled topics first allows all of the topics that are in the allow list, and then removes topics that are on the block list.By default, all topics are enabled (the allow list value is *
) and no topics are disabled (the block list has no value).
Use Web Console or a sling:OsgiConfig
node to configure the following properties. For sling:OsgiConfig
nodes, the PID of the Job Consumer Manager service is org.apache.sling.event.impl.jobs.JobConsumerManager.
Property Name in Web Console | OSGi ID | Description |
---|---|---|
Topic Whitelist | job.consumermanager.whitelist | A list of topics that the local JobManager service processes. The default value of * causes all topics to be sent to the registered TopicConsumer service. |
Topic Blacklist | job.consumermanager.blacklist | A list of topics that the local JobManager service does not process. |
The offloading framework uses replication to transport resources between author and worker. The offloading framework automatically creates replication agents when instances join the topology. The agents are created with default values. You must manually change the password that the agents use for authentication.
A known issue with the automatically-generated replication agents requires you to manually create new replication agents. Follow the procedure in Problems Using the Automatically Generated Replication Agents before you create the agents for Offloading.
Create the replication agents that transport job payloads between instances for offloading. The following illustration shows the agents that are required to offload from the author to a worker instance. The author has a Sling ID of 1 and the worker instance has a Sling ID of 2:
This setup requires the following three agents:
This replication scheme is similar to that used between author and publish instances. However, for the offloading situation all of the instances involved are authoring instances.
The Offloading framework uses the topology to obtain the IP addresses of the offloading instances. The framework then automatically creates the replication agents based on these IP addresses. If the IP addresses of the offloading instances later change, the change is automatically propaged on the topology after the instance restarts. However, the Offloading framework does not automatically update the replication agents to reflect the new IP addresses. To avoid this situaion, use fixed IP addresses for all instances in the topology.
Use a specific format for the Name property of the replication agents so that the offloading framework automatically uses the correct agent for specific worker instances.
Naming the outgoing agent on the author instance:
offloading_<slingid>
, where <slingid>
is the Sling ID of the worker instance.
Example: offloading_f5c8494a-4220-49b8-b079-360a72f71559
Naming the reverse agent on the author instance:
offloading_reverse_<slingid>
, where <slingid>
is the Sling ID of the worker instance.
Example: offloading_reverse_f5c8494a-4220-49b8-b079-360a72f71559
Naming the outbox on the worker instance:
offloading_outbox
Create a Replication Agent on author. (See the documention for replication agents). Specify any Title. The Name must follow the naming convention.
Create the agent using the following properties:
Property | Value |
---|---|
Settings > Serialization Type | Default |
Transport >Transport URI | https://<ip of target instance> :<port> /bin/receive?sling:authRequestLogin=1 |
Transport >Transport User | Replication user on target instance |
Transport >Transport Passoword | Replication user password on target instance |
Extended > HTTP Method | POST |
Triggers > Ignore Default | True |
Create a Reverse Replication Agent on author. (See the documention for replication agents.) Specify any Title. The Name must follow the naming convention.
Create the agent using the following properties:
Property | Value |
---|---|
Settings > Serialization Type | Default |
Transport >Transport URI | https://<ip of target instance> :<port> /bin/receive?sling:authRequestLogin=1 |
Transport >Transport User | Replication user on target instance |
Transport >Transport Passoword | Replication user password on target instance |
Extended > HTTP Method | GET |
Create a Replication Agent on the worker instance. (See the documention for replication agents.) Specify any Title. The Name must be offloading_outbox
.
Create the agent using the following properties.
Property | Value |
---|---|
Settings > Serialization Type | Default |
Transport >Transport URI | repo://var/replication/outbox |
Trigger > Ignore Default | True |
Obtain the Sling ID of an Experience Manager instance using either of the following methods:
Configure the instances of a topology so that specific instances perform the background processing of assets that are added or updated in DAM.
By default, Experience Manager executes the DAM Update Asset workflow when a DAM asset changes or one is added to DAM. Change the default behavior so that Experience Manager instead executes the DAM Update Asset Offloader workflow. This workflow generates a JobManager job that has a topic of com/adobe/granite/workflow/offloading
. Then, configure the topology so that the job is offloaded to a dedicated worker.
No workflow should be transient when used with workflow offloading. For example, the DAM Update Asset workflow must not be transient when used for asset offloading. To set/unset the transient flag on a workflow, see Transient Workflows.
The following procedure assumes the following characteristics for the offloading topology:
On each Experience Manager instance, configure the Discovery Service so that it points to the root Topography Connector. (See Configuring Topology Membership.)
Configure the root Topography Connector so that the connecting instances are on the allow list.
Open Offloading Browser and disable the com/adobe/granite/workflow/offloading
topic on the instances with which users interact to upload or change DAM assets.
On each instance that users interact with to upload or change DAM assets, configure workflow launchers to use the DAM Update Asset Offloading workflow:
On the instances that perform the background processing of DAM assets, disable the workflow launchers that execute the DAM Update Asset workflow.
In addition to the details presented on this page, you can also read the following: