Revision Cleanup

The persistence format of TarMK will change with AEM 6.5. These changes do not require a proactive migration step. Existing repositories will go through a rolling migration, which is transparent to the user. The migration process is initiated the first time AEM 6.5 (or related tools) access the repository.

Once the migration to the AEM 6.5 persistence format has been initiated the repository can not be reverted back to the previous AEM 6.3 persistence format.

In AEM 6.3 changes to the storage format were needed, especially for improving the performance and efficacy of Online Revision Cleanup. These changes are not backwards compatible, and repositories created with the old Oak Segment (AEM 6.2 and previous) must be migrated.

Additional benefits of changing the storage format:

• Better scalability (optimized segment size).
• Faster Data Store Garbage Collection.
  
• Ground work for future enhancements.

Offline Revision Cleanup is reclaiming everything but the latest generation compared to latest two generations for Online Revision Cleanup. In the case of a fresh repository, Online Revision Cleanup will not reclaim any space when executed for the first time after the Offline Revision Cleanup because there is no generation old enough to be reclaimed.

Aditionally, read the "Running Online Revision Cleanup after Offline Revision Cleanup" section of this chapter.

Online Revision Cleanup is available only with AEM 6.3 and higher releases. Also, if you are using an older version of AEM you need to migrate to the new Oak Segment Tar.

• Repository size
• Load on the system (requests per minute, specifically write operations)
• Activity pattern (reads versus writes)
• Hardware specifications (CPU performance, Memory, IOPS)

Disk space is continuously monitored during Online Revision Cleanup. Should the available disk space drop below a critical value, the process will be cancelled. The critical value is 25% of the current disk footprint of the repository and it is not configurable.

It is recommended to size the disk at least two or three times larger than the initially estimated repository size.

Free heap space is continuously monitored during the cleanup process. Should the free heap space drop below a critical value, the process is cancelled. The critical value is configured through org.apache.jackrabbit.oak.segment.SegmentNodeStoreService#MEMORY_THRESHOLD. The default value is 15%.

Recommendations for minimum compaction heap sizing are not separated from the AEM memory sizing recommendations. As a general rule: If an AEM instance is sized well enough to cope with the use cases and expected payload thereon, the cleanup process will obtain enough memory.

• Ensure that it is executed daily.
  
• Ensure that it is executed during minimal repository activities by configuring the maintenance windows in Operations Dashboard accordingly.
• Scale up system resources (CPU, Memory, I/O).

Revision Cleanup relies on an estimation phase to decide if there is enough garbage to be cleaned. The estimator compares the current size against the size of the repository after it was last compacted. If the size exceeds the configured delta, cleanup will run. The size delta is set at 1 GB. This effectively means that if the repository size did not grow by 1 GB since the last cleanup run, the new revision cleanup iteration will be skipped.

Below are the relevant log entries for the estimation phase:

• Revision GC will run: Size delta is N% or N/N (N/N bytes), so running compaction
• Revision GC will not run: Size delta is N% or N/N (N/N bytes), so skipping compaction for now

If there's write concurrency on the system, online revision cleanup might require exclusive write access to be able to commit the changes at the end of a compaction cycle. The system will go into forceCompact mode, as explained in more detail in the oak documentation. During force compact, an exclusive write lock is acquired in order to finally commit the changes without any concurrent writes interfering. To limit the impact on response times a time out value can be defined. This value is set to 1 minute by default, which means that if force compact does not complete within 1 minute the compaction process will be aborted in favour of concurrent commits.

The duration of force compact depends on the following factors:

• hardware: specifically IOPS. The duration decreases with more IOPS.
• segment store size: duration increases with the size of the segment store.

How is Online Revision Cleanup executed on a standby instance?

In a cold standby setup, only the primary instance needs to be configured to run Online Revision Cleanup. On the standby instance, Online Revision Cleanup does not need to be scheduled specifically.

The corresponding operation on a standby instance is the Automatic Cleanup - this corresponds to the cleanup phase of the Online Revision Cleanup. The Automatic Cleanup is run on the standby instance after the execution of the Online Revision Cleanup on the primary instance.

Estimation and compaction phases will not be run on a standby instance.

Offline Revision Cleanup can immediately remove old revisions while Online Revision Cleanup needs to account for old revisions still being referenced by the application stack. The former can thus remove garbage more aggressively than the latter where the effect is amortised over the course of a few garbage collection cycles.

Aditionally, read the "Running Online Revision Cleanup after Offline Revision Cleanup" section of this chapter.

• On Windows environments, regular file access is always enforced so memory mapped access is not used. As a general advice, all the available RAM should be allocated to the heap and the segmentCache size should be increased. You increase the segmentCache by adding the segmentCache.size option to the org.apache.jackrabbit.oak.segment.SegmentNodeStoreService.config (for example, segmentCache.size=20480). Remember to leave out some RAM for the operating system and other processes.
• On non Windows environments, increase the size of the physical memory to improve the memory mapping of the repository.

• Disk space should be monitored when Online Revision Cleanup is enabled. The cleanup will not run or it will terminate preemptively when there is insufficient disk space.
• Check the logs for the completion time of the Online Revision Cleanup. It should not take longer than 2 hours.
• Number of checkpoints. If there are more than 3 checkpoints when compaction runs it is recommended to clean up the checkpoints.

You can check if the Online Revision Cleanup has completed successfully by checking the logs.

For example, "TarMK GC #{}: compaction completed in {} ({} ms), after {} cycles" means the compaction step completed successfully unless preceded by the message "TarMK GC #{}: compaction gave up compacting concurrent commits after {} cycles", which means there was too much concurrent load.

Correspondingly there is a message "TarMK GC #{}: cleanup completed in {} ({} ms" for the successful completion of the cleanup step.

Status, progress and statistics are exposed via JMX (SegmentRevisionGarbageCollection MBean). For more details about the SegmentRevisionGarbageCollection MBean, read the following paragraph.

Progress can be tracked via the EstimatedRevisionGCCompletion attribute of the SegmentRevisionGarbageCollection MBean.

You can obtain a reference of the MBean using the ObjectName org.apache.jackrabbit.oak:name="Segment node store revision garbage collection",type="SegmentRevisionGarbageCollection".

Note that the statistics are only available since the last system start. External monitoring tooling could be leveraged to keep the data beyond AEM uptime. See the AEM documentation for attaching health checks to Nagios as an example for an external monitoring tool.

• Online Revision Cleanup has started / stopped
  • Online Revision Cleanup is composed of three phases: estimation, compaction and cleanup. Estimation can force compaction and cleanup to skip if the repository does not contain enough garbage. In the latest version of AEM, the message "TarMK GC #{}: estimation started" marks the start of estimation, "TarMK GC #{}: compaction started, strategy={}" marks the start of compaction and "TarMK GC #{}: cleanup started. Current repository size is {} ({} bytes" marks the start of cleanup.
• Disk space gained by the revision cleanup
  • Space is reclaimed only when the cleanup phase completes. The completion of the cleanup phase is marked by the log message "TarMK GC #{}: cleanup completed in {} ({} ms". Post cleanup size is {} ({} bytes) and space reclaimed {} ({} bytes). Compaction map weight/depth is {}/{} ({} bytes/{}).".
• A problem occured during the revision cleanup
  • There are many failure conditions, all of them are marked by WARN or ERROR log messages staring with "TarMK GC".

Also, see the Troubleshooting Based on Error Messages section below.

A repository integrity check is not needed after the Online Revision Cleanup.

However, you can perform the following actions to check the repository status after cleanup:

• A repository traversal check
• Use the oak-run tool after the cleanup process has completed to check for inconsistencies. For further info on how to do this, check the Apache Documentation. You do not need to shut down AEM to run the tool.

The Revision Clean-up Health Check is part of the Operations Dashboard.

The status will be GREEN if the last execution of the Online Revision Cleanup maintenance task has completed succesfully.

It will be YELLOW if the Online Revision Cleanup maintenance task was cancelled once.

It will be RED if the Online Revision Cleanup maintenance task was cancelled three times in a row. In this case manual interaction is required or Online Revision Clean-up is likely to fail again. For more information, read the Troubleshooting section below.

Also note that the Health Check status will be reset after a system restart. So a freshly restarted instance will show a green status on the Revision Cleanup Health Check. External monitoring tooling could be leveraged to keep the data beyond AEM uptime. See the AEM documentation for attaching health checks to Nagios as an example for an external monitoring tool.

How to monitor Automatic Cleanup on a standby instance?

Status, progress and statistics are exposed via JMX by using the SegmentRevisionGarbageCollection MBean. See also the following Oak documentation.

You can obtain a reference of the MBean by using the ObjectName org.apache.jackrabbit.oak:name="Segment node store revision garbage collection",type="SegmentRevisionGarbageCollection".

Note that the statistics are available only since the last system start. External monitoring tooling could be leveraged to keep the data beyond the AEM uptime. Also, see See the AEM documentation for attaching health checks to Nagios as an example for an external monitoring tool.

The log files can also be used to check the status, progress and statistics of the Automatic Cleanup.

What needs to be monitored during Automatic Cleanup on a standby instance?

• Disk space should be monitored when the Automatic Cleanup is run.
• Completion time (via the logs) to ensure that 2 hours are not exceeded.
• Segmentstore size after the Automatic Cleanup has run. The size of the segmentstore on the standby instance should be approximately the same as the one on the primary instance.

• First, check the log entries
  
• Depending on the information in the logs, take appropriate action:
  • If the logs show five missed compact cycles and a timeout on the forceCompact cycle, schedule the maintenance window to a quiet time when the amount of repository writes is low. You can check repository writes in the repositoy metrics monitoring tool located at https://serveraddress:serverport/libs/granite/operations/content/monitoring/page.html
  • If the cleanup stopped at the end of the maintenance window, make sure the configuration of the maintenance window in the Maintenance Tasks user interface is big enough
  • If available heap memory is not sufficient, make sure instance has enough memory.
  • In case of a late reaction, the segmentstore might grow too much for Online Revision Cleanup to complete even within a longer maintenance window. For example, if there was no successful Online Revision Cleanup completed in the last week then it is recommended to plan an offline maintenance and to execute Offline Revision Cleanup in order to bring the segmenstore back to a manageable size.

A SegmentNotFoundException is logged by the TarMK when it tries to access a storage unit (a segment) that it can not find. There are three scenarios that could cause this issue:

1. An application that circumvents the recommended access mechanisms (like Sling and the JCR API) and uses a lower level API/SPI to access the repository and then exceeds the retention time of a segment. That is, it keeps a reference to an entity longer than the retention time allowed by the Online Revision Cleanup (24 hours by default). This case is transient and doesn't lead to data corruption. To recover, the oak-run tool should be used to confirm the transient nature of the exception (the oak-run check should not report any errors). In order to do this the instance needs to be taken offline and restarted afterwards.
2. An external event caused the corruption of the data on the disk. This can be a disk failure, out of disk space or an accidental modification of the required data files. In this case, the instance needs to be taken offline and repaired using the oak-run check. For more details on how to perform the oak-run check, read the following Apache documentation.
3. All other occurrences should addressed through the Adobe Customer Care.

The repository size and the amount of revisions that need to be cleaned up determines the duration of the cleanup.

• Oak revision: Oak organizes all the content in a large tree hierarchy that consists of nodes and properties. Each snapshot or revision of this content tree is immutable, and changes to the tree are expressed as a sequence of new revisions. Typically, each content modification triggers a new revision. See also Follow link.
• Page Version: Versioning creates a "snapshot" of a page at a specific point in time. Typically, a new version is created when a page is activated. For more information, see Working with Page Versions.