Operations Dashboard

Query Performance

This health check was simplified in AEM 6.4, and now checks the recently-refactored Oak QueryStats MBean, more specifically the SlowQueries attribute. If the statistics contain any slow queries, then the health check returns a warning. Otherwise, it returns the OK status.

The MBean for this health check is org.apache.sling.healthcheck:name=queriesStatus,type=HealthCheck.

Observation Queue Length

Observation Queue Length iterates over all Event Listeners and Background Observers, compares their queueSize to their maxQueueSize and:

• returns Critical status if the queueSize value exceeds the maxQueueSize value (that is when events would be dropped)
• returns Warn if the queueSize value is over the maxQueueSize * WARN_THRESHOLD (the default value is 0.75)

The maximum length of each queue comes from separate configurations (Oak and AEM), and is not configurable from this health check. The MBean for this health check is org.apache.sling.healthcheck:name=ObservationQueueLengthHealthCheck,type=HealthCheck.

Query Traversal Limits

Query Traversal Limits checks the QueryEngineSettings MBean, more specifically the LimitInMemory and LimitReads attributes, and returns the following status:

• returns the Warn status if one of the limits is equal or higher than the Integer.MAX_VALUE
• returns the Warn status if one of the limits is lower than 10000 (the recommended setting from Oak)
• returns the Critical status if the QueryEngineSettings or any of the limits cannot be retrieved

The Mbean for this health check is org.apache.sling.healthcheck:name=queryTraversalLimitsBundle,type=HealthCheck.

Synchronized Clocks

This check is relevant only for document nodestore clusters. It returns the following status:

• returns the Warn status when the instance clocks get out of sync and go over a predefined low threshold
• returns the Critical status when the instance clocks get out of sync and go over a predefined high threshold

The Mbean for this health check is org.apache.sling.healthcheck:name=slingDiscoveryOakSynchronizedClocks,type=HealthCheck.

Asynchronous Indexes

The Asynchronous Indexes check:

• returns Critical status if at least one indexing lane is failing

• checks the lastIndexedTime for all indexing lanes and:

  • returns Critical status if it's more than 2 hours ago
  • returns Warning status if it's between 2 hours and 45 minutes ago
  • returns OK status if it's less than 45 minutes ago

• if none of these conditions are met, it returns the OK status

Both the Critical and Warn status thresholds are configurable. The Mbean for this health check is org.apache.sling.healthcheck:name=asyncIndexHealthCheck,type=HealthCheck.

Note: This health check is available with AEM 6.4 and has been backported to AEM 6.3.0.1.

Large Lucene Indexes

This check uses the data exposed by the Lucene Index Statistics MBean to identify large indexes and returns:

• a Warning status if there is an index with more than 1 billion documents
• a Critical status if there is an index with more than 1.5 billion documents

The thresholds are configurable and the MBean for the health check is org.apache.sling.healthcheck:name=largeIndexHealthCheck,type=HealthCheck.

Note: This check is available with AEM 6.4 and has been backported to AEM 6.3.2.0.

System Maintenance

System Maintenance is a composite check that returns the OK if all maintenance tasks are running as configured. Keep in mind that:

• each maintenance task is accompanied by an associated health check
• if a task is not added to a maintenance window, its health check will return Critical
• you need to configure the Audit Log and Workflow Purge maintenance tasks or otherwise remove them from the maintenance windows. If left unconfigured, these tasks will fail on the first attempted run, so the System Maintenance check will return the Critical status.
• With AEM 6.4, there is also a check for the Lucene Binaries Maintenance task
• on AEM 6.2 and lower, the system maintenance check returns a Warning status right after startup because the tasks never run. Starting with 6.3, they will return OK if the first maintenance window wasn't reached yet.

The MBean for this health check is org.apache.sling.healthcheck:name=systemchecks,type=HealthCheck.

Replication Queue

This check iterates over replication agents and looks at their queues. For the item at the top of the queue, the check looks at how many times the agent retried replication. If the agent retried replication more than the value of the numberOfRetriesAllowed parameter, it returns a warning. The numberOfRetriesAllowed parameter is configurable.

The MBean for this health check is org.apache.sling.healthcheck:name=replicationQueue,type=HealthCheck.

Sling Jobs

Sling Jobs checks the number of jobs queued in the JobManager, compares it to the maxNumQueueJobs threshold, and:

• returns Critical if more than the maxNumQueueJobs are in the queue
• returns Critical if there are long-running active jobs that are older than 1 hour
• returns Critical if there are queued jobs, and the last finished job time is older than 1 hour

Only the maximum number of queued jobs parameter is configurable and it has the default value of 1000.

The MBean for this health check is org.apache.sling.healthcheck:name=slingJobs,type=HealthCheck.

Request Performance

This check looks at the granite.request.metrics.timer Sling metricand:

• returns Critical if the 75th percentile value is over the critical threshold (the default value is 500 milliseconds)
• returns Warn if the 75th percentile value is over the warn threshold (the default value is 200 milliseconds)

The MBean for this health check is org.apache.sling.healthcheck:name=requestsStatus,type=HealthCheck.

Log Errors

This check returns the Warn status if there are errors in the log.

The MBean for this health check is org.apache.sling.healthcheck:name=logErrorHealthCheck,type=HealthCheck.

Disk Space

The Disk Space check looks at the FileStoreStats MBean, retrieves the size of the Node Store and the amount of usable disk space on the Node Store partition, and:

• returns Warn if the usable disk space to repository size ratio is less than the warn threshold (the default value is 10)
• returns Critical if the usable disk space to repository size ratio is less than the critical threshold (the default value is 2)

Both thresholds are configurable. The check only works on instances with a Segment Store.

The MBean for this health check is org.apache.sling.healthcheck:name=DiskSpaceHealthCheck,type=HealthCheck.

Scheduler Health Check

This check returns a warning if the instance has Quartz jobs running for more than 60 seconds. The acceptable duration threshold is configurable.

The MBean for this health check is org.apache.sling.healthcheck:name=slingCommonsSchedulerHealthCheck,type=HealthCheck.

Security Checks

The Security check is a composite which aggregates the results of multiple security-related checks. These individual health checks address different concerns from the security checklist available at the Security Checklist documentation page. The check is useful as a security smoke test when the instance is started.

The MBean for this health check is org.apache.sling.healthcheck:name=securitychecks,type=HealthCheck

Active Bundles

Active Bundles checks the state of all bundles and:

• returns the Warn status if any of the bundles is not active or (starting, with lazy activation)
• it ignores the status of bundles in the ignore list

The ignore list parameter is configurable.

The MBean for this health check is org.apache.sling.healthcheck:name=inactiveBundles,type=HealthCheck.

Code Cache Check

This is a Health Check that verifies several JVM conditions that can trigger a CodeCache bug present in Java 7:

• returns Warn if the instance is running on Java 7, with Code Cache flushing enabled
• returns Warn if the instance is running on Java 7, and the Reserved Code Cache size is less than a minimum threshold (the default value is 90MB)

The minimum.code.cache.size threshold is configurable. For more information about the bug, check this page.

The MBean for this health check is org.apache.sling.healthcheck:name=codeCacheHealthCheck,type=HealthCheck.

Resource Search Path Errors

Checks if there are any resources in the path /apps/foundation/components/primary and:

• returns Warn if there are child nodes under /apps/foundation/components/primary

The MBean for this health check is org.apache.sling.healthcheck:name=resourceSearchPathErrorHealthCheck,type=HealthCheck.

granite.maintenance.isStoppable

Boolean attribute defining whether the task can be stopped by the user. If a task declares that it is stoppable it must check during its execution whether it has been stopped and then act accordingly. The default is false.

true

Optional

granite.maintenance.mandatory

Boolean attribute defining whether a task is mandatory and must be run periodically. If a task is mandatory but currently not in any active schedule window, a Health Check will report this as an error. The default is false.

true

Optional

granite.maintenance.name

A unique name for the task - this is used to reference the task. This is usally a simple name.

MyMaintenanceTask

Required

granite.maintenance.title

A title displayed for this task

My Special Maintenance Task

Required

job.topics

This is a unique topic of the maintenance task.
The Apache Sling job handling will start a job with exactly this topic to execute the maintenance task and as the task is registered for this topic it gets executed.
The topic must start with com/adobe/granite/maintenance/job/

com/adobe/granite/maintenance/job/MyMaintenanceTask

Required

/*

* #%L

* sample-maintenance-task

* %%

* Copyright (C) 2014 Adobe

* %%

* Licensed under the Apache License, Version 2.0 (the "License");

* you may not use this file except in compliance with the License.

* You may obtain a copy of the License at

*

* https://www.apache.org/licenses/LICENSE-2.0

*

* Unless required by applicable law or agreed to in writing, software

* distributed under the License is distributed on an "AS IS" BASIS,

* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.

* See the License for the specific language governing permissions and

* limitations under the License.

* #L%

*/

``

package com.adobe.granite.samples.maintenance.impl;

``

import java.io.File;

import java.util.Calendar;

import java.util.Collection;

import java.util.Map;

``

import org.apache.commons.io.FileUtils;

import org.apache.commons.io.filefilter.IOFileFilter;

import org.apache.commons.io.filefilter.TrueFileFilter;

import org.apache.felix.scr.annotations.Activate;

import org.apache.felix.scr.annotations.Component;

import org.apache.felix.scr.annotations.Properties;

import org.apache.felix.scr.annotations.Property;

import org.apache.felix.scr.annotations.Service;

import org.apache.sling.commons.osgi.PropertiesUtil;

import org.apache.sling.event.jobs.Job;

import org.apache.sling.event.jobs.consumer.JobConsumer;

import org.apache.sling.event.jobs.consumer.JobExecutionContext;

import org.apache.sling.event.jobs.consumer.JobExecutionResult;

import org.apache.sling.event.jobs.consumer.JobExecutor;

import org.slf4j.Logger;

import org.slf4j.LoggerFactory;

``

import com.adobe.granite.maintenance.MaintenanceConstants;

``

@Component(metatype = true,

label = "Delete Temp Files Maintenance Task",

description = "Maintatence Task which deletes files from a configurable temporary directory which have been modified in the last 24 hours.")

@Service

@Properties({

@Property(name = MaintenanceConstants.PROPERTY_TASK_NAME, value = "DeleteTempFilesTask", propertyPrivate = true),

@Property(name = MaintenanceConstants.PROPERTY_TASK_TITLE, value = "Delete Temp Files", propertyPrivate = true),

@Property(name = JobConsumer.PROPERTY_TOPICS, value = MaintenanceConstants.TASK_TOPIC_PREFIX

+ "DeleteTempFilesTask", propertyPrivate = true) })

public class DeleteTempFilesTask implements JobExecutor {

``

private static final Logger log = LoggerFactory.getLogger(DeleteTempFilesTask.class);

``

@Property(label = "Temporary Directory", description="Temporary Directory. Defaults to the java.io.tmpdir system property.")

private static final String PROP_TEMP_DIR = "temp.dir";

``

private File tempDir;

``

@Activate

private void activate(Map<string, object=""> properties) {

this.tempDir = new File(PropertiesUtil.toString(properties.get(PROP_TEMP_DIR),

System.getProperty("java.io.tmpdir")));

}

``

@Override

public JobExecutionResult process(Job job, JobExecutionContext context) {

log.info("Deleting old temp files from {}.", tempDir.getAbsolutePath());

Collection<file> files = FileUtils.listFiles(tempDir, new LastModifiedBeforeYesterdayFilter(),

TrueFileFilter.INSTANCE);

int counter = 0;

for (File file : files) {

log.debug("Deleting file {}.", file.getAbsolutePath());

counter++;

file.delete();

// TODO - capture the output of delete() and do something useful with it

}

return context.result().message(String.format("Deleted %s files.", counter)).succeeded();

}

``

/**

* IOFileFilter which filters out files which have been modified in the last 24 hours.

*

*/

private static class LastModifiedBeforeYesterdayFilter implements IOFileFilter {

``

private final long minTime;

``

private LastModifiedBeforeYesterdayFilter() {

Calendar cal = Calendar.getInstance();

cal.add(Calendar.DATE, -1);

this.minTime = cal.getTimeInMillis();

}

``

@Override

public boolean accept(File dir, String name) {

// this method is never actually called.

return false;

}

``

@Override

public boolean accept(File file) {

return file.lastModified() <= this.minTime;

}

}

``

}

<file></string,>