Revision Cleanup

Questions

Answers

What should I be aware of when I upgrade to AEM 6.5?

The persistence format of TarMK changes with AEM 6.5. These changes do not require a proactive migration step. Existing repositories go through a rolling migration, which is transparent to the user. The migration process is initiated the first time AEM 6.5 (or related tools) access the repository.

Once the migration to the AEM 6.5 persistence format has been initiated, the repository cannot be reverted to the previous AEM 6.3 persistence format.

In AEM 6.3 changes to the storage format were needed, especially for improving the performance and efficacy of Online Revision Cleanup. These changes are not backwards compatible, and repositories created with the old Oak Segment (AEM 6.2 and previous) must be migrated.

Additional benefits of changing the storage format:

• Better scalability (optimized segment size).
• Faster Data Store Garbage Collection.
• Ground work for future enhancements.

Only the new Oak Segment Tar is supported with AEM 6.3 or higher.

Yes. Unless you start with a fresh instance, you will always have to migrate the content.

No, as explained above, the content migration is mandatory.

No. This is a one time effort that cannot be done on a running instance.

If you try to run the oak-segment module against an oak-segment-tar repository (or conversely), startup fails with an IllegalStateException with the message "Invalid segment format". No data corruption occurs.

No. Migrating from oak-segment to oak-segment-tar introduces changes in the container format. The contained data is not affected and will not be modified.

The migration is equivalent to recreating the segmentstore in the new format. This can be used to estimate the additional disk space needed during migration. After the migration, the old segment store can be deleted to reclaim space.

Migration performance can be greatly improved if offline revision cleanup is executed prior to the migration. All customers are advised to execute it as a pre-requisite of the upgrade process. In general, the duration of the migration should be similar to the duration of the offline revision cleanup task, assuming that the offline revision cleanup task has been executed before the migration.

Once per day. This is the default configuration in the Operations Dashboard.

See the How to run Online Revision Cleanup section.

It is recommended to run Online Revision Cleanup once per day, as configured by default.

There is no need to determine the frequency as Online Revision Cleanup is configured as a maintenance task and it automatically runs each day.

Online Revision Cleanup reclaims old revisions by generations. A fresh generation is generated every time revision cleanup runs. Only the content that is at least two generations old will be reclaimed, which means that on a first run there is nothing to reclaim.

Offline Revision Cleanup is reclaiming everything but the latest generation compared to latest two generations for Online Revision Cleanup. If there is a fresh repository, Online Revision Cleanup will not reclaim any space when executed for the first time after the Offline Revision Cleanup because there is no generation old enough to be reclaimed.

Also, read the "Running Online Revision Cleanup after Offline Revision Cleanup" section of this chapter.

This depends on office hours and the traffic patterns of the customer online presence. The maintenance windows should be configured outside of the main production times to allow for the best cleanup efficacy. For multiple AEM Publish instances (TarMK Farm), maintenance windows for Online Revision Cleanup should be staggered.

Online Revision Cleanup is available only with AEM 6.3 and higher releases. Also, if you are using an older version of AEM, you must migrate to the new Oak Segment Tar.

The factors are:

• Repository size
• Load on the system (requests per minute, specifically write operations)
• Activity pattern (reads versus writes)
• Hardware specifications (CPU performance, Memory, IOPS)

Yes, Online Revision Cleanup can cope with concurrent writes. However, Online Revision Cleanup works faster and more efficiently without concurrent write transactions. Adobe recommends scheduling the Online Revision Cleanup maintenance task to a relatively quiet time without a lot traffic.

Disk space is continuously monitored during Online Revision Cleanup. Should the available disk space drop below a critical value, the process is canceled. The critical value is 25% of the current disk footprint of the repository and it is not configurable.

Adobe recommends you size the disk at least two or three times larger than the initially estimated repository size.

Free heap space is continuously monitored during the cleanup process. Should the free heap space drop below a critical value, the process is canceled. The critical value is configured through org.apache.jackrabbit.oak.segment.SegmentNodeStoreService#MEMORY_THRESHOLD. The default value is 15%.

Recommendations for minimum compaction heap sizing are not separated from the AEM memory sizing recommendations. Generally: If an AEM instance is sized enough to cope with the use cases and expected payload thereon, the cleanup process obtains enough memory.

Online Revision Cleanup is a background process that reads from and writes to the repository concurrently to normal system operations. In particular, it might need to acquire exclusive access to the repository for a short time period, preventing other threads from writing into the repository.

It should take no longer than two hours to be run according to the latest performance tests Adobe performed internally.

• Ensure that it is executed daily.
• Ensure that it is executed during minimal repository activities by configuring the maintenance windows in Operations Dashboard accordingly.
• Scale up system resources (CPU, Memory, I/O).

Make sure that other maintenance tasks are not delaying its execution. This could be the case if more maintenance tasks than Online Revision Cleanup are executed within the same maintenance window. Maintenance tasks are run sequentially without a configurable order.

Revision Cleanup relies on an estimation phase to decide if there is enough garbage to be cleaned. The estimator compares the current size against the size of the repository after it was last compacted. If the size exceeds the configured delta, cleanup runs. The size delta is set at 1 GB. This effectively means that if the repository size did not grow by 1 GB since the last cleanup run, the new revision cleanup iteration is skipped.

Below are the relevant log entries for the estimation phase:

• Revision GC runs: Size delta is N% or N/N (N/N bytes), so running compaction
• Revision GC does not run: Size delta is N% or N/N (N/N bytes), so skipping compaction for now

Yes. Since AEM 6.3, it can be safely stopped by way of the Maintenance Task Window within the Operations Dashboard or by way of JMX.

Revision Cleanup is interrupted and the repository shuts down safely.

There is no risk of data corruption in such cases. Garbage leftovers are cleaned up by a subsequent run.

Performance degradation over time.

By default, the Online Revision Cleanup only collects revisions that are at least 24 hours old.

If there's write concurrency on the system, online revision cleanup might require exclusive write access to be able to commit the changes at the end of a compaction cycle. The system goes into forceCompact mode, as explained in more detail in the Oak documentation. During force compact, an exclusive write lock is acquired to finally commit the changes without any concurrent writes interfering. To limit the impact on response times, a time-out value can be defined. This value is set to one minute by default, which means that if force compact does not complete within one minute, the compaction process is aborted in favor of concurrent commits.

The duration of force compact depends on the following factors:

• hardware: specifically IOPS. The duration decreases with more IOPS.
• segment store size: duration increases with the size of the segment store.

In a cold standby setup, only the primary instance must be configured to run Online Revision Cleanup. On the standby instance, Online Revision Cleanup does not need to be scheduled specifically.

The corresponding operation on a standby instance is the Automatic Cleanup - this corresponds to the cleanup phase of the Online Revision Cleanup. The Automatic Cleanup is run on the standby instance after the execution of the Online Revision Cleanup on the primary instance.

Estimation and compaction phases will not be run on a standby instance.

Offline Revision Cleanup can immediately remove old revisions while Online Revision Cleanup must account for old revisions still being referenced by the application stack. The former can thus remove garbage more aggressively than the latter where the effect is amortized over the course of a few garbage collection cycles.

Also, read the "Running Online Revision Cleanup after Offline Revision Cleanup" section of this chapter.

Any considerations about memory mapped file operations?

• On Windows environments, regular file access is always enforced so memory mapped access is not used. As a general advice, all the available RAM should be allocated to the heap and the segmentCache size should be increased. You increase the segmentCache by adding the segmentCache.size option to the org.apache.jackrabbit.oak.segment.SegmentNodeStoreService.config (for example, segmentCache.size=20480). Remember to leave out some RAM for the operating system and other processes.
• On non-Windows environments, increase the size of the physical memory to improve the memory mapping of the repository.

• Disk space should be monitored when Online Revision Cleanup is enabled. The cleanup does not run or it terminates preemptively when there is insufficient disk space.
• Check the logs for the completion time of the Online Revision Cleanup. It should not take longer than 2 hours.
• Number of checkpoints. If there are more than 3 checkpoints when compaction runs it is recommended to clean up the checkpoints.

You can check if the Online Revision Cleanup has completed successfully by checking the logs.

For example, "TarMK GC #{}: compaction completed in {} ({} ms), after {} cycles" means the compaction step completed successfully unless preceded by the message "TarMK GC #{}: compaction gave up compacting concurrent commits after {} cycles", which means there was too much concurrent load.

Correspondingly there is a message "TarMK GC #{}: cleanup completed in {} ({} ms" for the successful completion of the cleanup step.

Status, progress, and statistics are exposed via JMX (SegmentRevisionGarbageCollection MBean). For more details about the SegmentRevisionGarbageCollection MBean, read the following paragraph.

Progress can be tracked via the EstimatedRevisionGCCompletion attribute of the SegmentRevisionGarbageCollection MBean.

You can obtain a reference of the MBean using the ObjectName org.apache.jackrabbit.oak:name="Segment node store revision garbage collection",type="SegmentRevisionGarbageCollection".

The statistics are only available since the last system start. External monitoring tooling could be used to keep the data beyond AEM uptime.

• Online Revision Cleanup has started / stopped
  • Online Revision Cleanup is composed of three phases: estimation, compaction, and cleanup. Estimation can force compaction and cleanup to skip if the repository does not contain enough garbage. In the latest version of AEM, the message "TarMK GC #{}: estimation started" marks the start of estimation, "TarMK GC #{}: compaction started, strategy={}" marks the start of compaction and "TarMK GC #{}: cleanup started. Current repository size is {} ({} bytes" marks the start of cleanup.
• Disk space gained by the revision cleanup
  • Space is reclaimed only when the cleanup phase completes. The completion of the cleanup phase is marked by the log message "TarMK GC #{}: cleanup completed in {} ({} ms". Post cleanup size is {} ({} bytes) and space reclaimed {} ({} bytes). Compaction map weight/depth is {}/{} ({} bytes/{}).".
• A problem occurred during the revision cleanup
  • There are many failure conditions, all of them are marked by WARN or ERROR log messages staring with "TarMK GC".

Also, see the Troubleshooting Based on Error Messages section below.

There is a message in the log at the end of the cleanup cycle: "TarMK GC #3: cleanup completed" that includes the size of the repository and the amount of reclaimed garbage.

A repository integrity check is not needed after the Online Revision Cleanup.

However, you can perform the following actions to check the repository status after cleanup:

• A repository traversal check
• Use the oak-run tool after the cleanup process has completed to check for inconsistencies. For further info on how to do this, check the Apache Documentation. You do not need to shut down AEM to run the tool.

Failure conditions are marked by WARN or ERROR log messages starting with "TarMK GC". Also, see the Troubleshooting Based on Error Messages section below.

The Revision Clean-up Health Check is part of the Operations Dashboard.

The status is GREEN if the last execution of the Online Revision Cleanup maintenance task has completed successfully.

It is YELLOW if the Online Revision Cleanup maintenance task was canceled once.

It is RED if the Online Revision Cleanup maintenance task was canceled three times in a row. In this case manual interaction is required or Online Revision Clean-up is likely to fail again. For more information, read the Troubleshooting section below.

Also, the Health Check status is reset after a system restart. So, a freshly restarted instance shows a green status on the Revision Cleanup Health Check. External monitoring tooling could be used to keep the data beyond AEM uptime.

Status, progress, and statistics are exposed via JMX by using the SegmentRevisionGarbageCollection MBean. See also the following Oak documentation.

You can obtain a reference of the MBean by using the ObjectName org.apache.jackrabbit.oak:name="Segment node store revision garbage collection",type="SegmentRevisionGarbageCollection".

The statistics are available only since the last system start. External monitoring tooling could be used to keep the data beyond AEM uptime.

The log files can also be used to check the status, progress, and statistics of the Automatic Cleanup.

• Disk space should be monitored when the Automatic Cleanup is run.
• Completion time (via the logs) to ensure that 2 hours are not exceeded.
• Segmentstore size after the Automatic Cleanup has run. The size of the segmentstore on the standby instance should be approximately the same as the one on the primary instance.

The AEM instance runs out of disk space, which causes outages in production.

High user traffic impacts whether the compaction phase is able to successfully finish or not.

You can take several steps to find and fix the issue:

• First, check the log entries

• Depending on the information in the logs, take appropriate action:

  • If the logs show five missed compact cycles and a timeout on the forceCompact cycle, schedule the maintenance window to a quiet time when the amount of repository writes is low. You can check repository writes in the repository metrics monitoring tool at https://serveraddress:serverport/libs/granite/operations/content/monitoring/page.html
  • If the cleanup stopped at the end of the maintenance window, make sure the configuration of the maintenance window in the Maintenance Tasks user interface is large enough
  • If available heap memory is not sufficient, make sure that instance has enough memory.
  • If there is a late reaction, the segmentstore might grow too much for Online Revision Cleanup to complete even within a longer maintenance window. For example, if there was no successful Online Revision Cleanup completed in the last week then it is recommended to plan an offline maintenance and to run Offline Revision Cleanup to bring the segmenstore back to a manageable size.

See the previous point.

Online Revision Cleanup is canceled and the leftovers are removed. It starts again next time the maintenance window is scheduled.

A SegmentNotFoundException is logged by the TarMK when it tries to access a storage unit (a segment) that it cannot find. There are three scenarios that could cause this issue:

1. An application that circumvents the recommended access mechanisms (like Sling and the JCR API) and uses a lower-level API/SPI to access the repository and then exceeds the retention time of a segment. That is, it keeps a reference to an entity longer than the retention time allowed by the Online Revision Cleanup (24 hours by default). This case is transient and does not lead to data corruption. To recover, the oak-run tool should be used to confirm the transient nature of the exception (the oak-run check should not report any errors). To do this, the instance must be taken offline and restarted afterwards.
2. An external event caused the corruption of the data on the disk. This can be a disk failure, out of disk space or an accidental modification of the required data files. In this case, the instance must be taken offline and repaired using the oak-run check. For more details on how to perform the oak-run check, read the following Apache documentation.
3. Address all other occurrences through the Adobe Customer Care.