Data Store Garbage Collection

Last update: May 3, 2023

CREATED FOR:

  • Admin
CAUTION
AEM 6.4 has reached the end of extended support and this documentation is no longer updated. For further details, see our technical support periods. Find the supported versions here.

When a conventional WCM asset is removed, the reference to the underlying data store record may be removed from the node hierarchy, but the data store record itself remains. This unreferenced data store record then becomes “garbage” that need not be retained. In instances where a number of garbage assets exist, it is beneficial to get rid of them to preserve space and to optimize backup and filesystem maintenance performance.

For the most part, a WCM application tends to collect information but not delete information nearly as often. Although new images are added, even superseding old versions, the version control system still retains the old one and supports reverting to it if needed. Thus the majority of the content we think of as adding to the system is effectively permanently stored. So what is the typical source of “garbage” in the repository that we might want to clean up?

AEM uses the repository as the storage for a number of internal and housekeeping activities:

  • Packages built and downloaded
  • Temporary files created for publish replication
  • Workflow payloads
  • Assets created temporarily during DAM rendering

When any of these temporary objects is large enough to require storage in the data store, and when the object eventually passes out of use, the data store record itself remains as “garbage”. In a typical WCM author/publish application, the largest source of garbage of this type is commonly the process of publish activation. When data is being replicated to Publish, it if first gathered in collections in an efficient data format called “Durbo” and stored in the repository under /var/replication/data. The data bundles are often larger than the critical size threshold for the data store and therefore wind up stored as data store records. When the replication is complete, the node in /var/replication/data is deleted, but the data store record remains as “garbage”.

Another source of recoverable garbage is packages. Package data, like everything else, is stored in the repository and thus for packages which are larger than 4KB, in the data store. In the course of a development project or over time while maintaining a system, packages may be built and rebuilt many times, each build resulting in a new data store record, orphaning the previous build’s record.

How does data store garbage collection work?

If the repository has been configured with an external data store, data store garbage collection will run automatically as part of the Weekly Maintenance Window. The system administrator can also run data store garbage collection manually on as as-needed basis. In general, it is recommended that data store garbage collection be performed periodically, but that the following factors be taken into account in planning data store garbage collections:

  • Data store garbage collections take time and may impact performance, so they should be planned accordingly.
  • Removal of data store garbage records does not affect normal performance, so this is not a performance optimization.
  • If storage utilization and related factors like backup times are not a concern, then data store garbage collection might be safely deferred.

The data store garbage collector first makes a note of the current timestamp when the process begins. The collection is then carried out using a multi-pass mark/sweep pattern algorithm.

In the first phase, the data store garbage collector performs a comprehensive traversal of all of the repository content. For each content object that has a reference to a data store record, it located the file in the filesystem, performing a metadata update – modifying the “last modified” or MTIME attribute. At this point files that are accessed by this phase become newer than the initial baseline timestamp.

In the second phase, the data store garbage collector traverses the physical directory structure of the data store in much the same way as a “find”. It examined the “last modified” or MTIME attribute of the file and makes the following determination:

  • If the MTIME is newer than the initial baseline timestamp, then either the file was found in the first phase, or it is an entirely new file that was added to the repository while the collection process was ongoing. In either of these cases the record is taken to be active and the file shall not be deleted.
  • If the MTIME is prior to the initial baseline timestamp, then the file is not an actively referenced file and it is considered removable garbage.

This approach works well for a single node with a private data store. However the data store may be shared, and if it is this means that potentially active live references to data store records from other repositories are not checked, and active referenced files may be mistakenly removed. It is imperative that the system admin understand the shared nature of the data store before planning any garbage collections, and only use the simple built-in data store garbage collection process when it is known that the data store is not shared.

NOTE
When performing garbage collection in a clustered or shared data store setup (with Mongo or Segment Tar) the log might display warnings about the inability to delete certain blob IDs. This happens because blob IDs deleted in a previous garbage collection are incorrectly referenced again by other cluster or shared nodes which do not have information about the ID deletions. As a result, when garbage collection is performed it logs a warning when it tries to delete an ID that has already been deleted in the last run. This behaviour does not affect performance or functionality.