AEM On-Premise: Segment store cleanup failure due to insufficient disk space

✨ This content is AI generated and could be inaccurate. Generative AI Terms

This article explains why automatic revision cleanup fails in Adobe Experience Manager (AEM) On-Premise environments when the segment store approaches the filesystem limit, and outlines steps to resolve the issue.

Description description

Environment

Adobe Experience Manager (AEM) 6.5.21.0 with Apache Jackrabbit Oak 1.22.20 on publish servers running Linux.

Symptoms

  • The segment store size increases to near the filesystem limit (e.g., 421–425 GB on a 550 GB filesystem).
  • Automatic revision cleanup (TarMK compaction) fails and does not run due to insufficient available disk space.
  • Repository size continues to grow, with no additional storage available for the segment store directory.
  • Attempts to run offline compaction are blocked by lack of disk headroom.
  • Error logs show repeated compaction cancellations, such as:
  • “TarMK GC #: compaction cancelled: not enough disk space.”
  • The cleanup process reports reclaiming 0 B, and the segment store continues to increase in size.
  • Disk usage commands (e.g., du -sh repository/segmentstore) confirm the segment store size.

Steps to reproduce

  1. Allow the segment store to grow until it approaches the filesystem limit.
  2. Attempt to run automatic or offline revision cleanup.
  3. Observe compaction failures due to insufficient disk space.

Cause

Automatic revision cleanup fails because Oak cancels TarMK tail compaction when there is not enough disk space to create a new segment generation. Without successful compaction, obsolete segment data cannot be reclaimed, causing the segment store to grow until disk space is exhausted.

Resolution resolution

Follow the steps below to resolve the issue:

  1. Do not manually delete files from repository/segmentstore while AEM is running, as this can cause repository corruption.

  2. Do not run offline compaction without a recent, restorable backup and sufficient temporary disk space.

  3. Increase filesystem capacity or provide temporary storage sufficient for the compaction working set.

  4. If permanent expansion is not possible, work with your infrastructure team to mount a temporary volume at the repository location or relocate the repository to a volume with adequate space during a maintenance window.

  5. Validate that the temporary volume has enough usable capacity and performance for the full compaction operation.

  6. Test the mount or relocation plan in a non-production environment if possible.

  7. Take and validate a backup of the AEM instance before any compaction or cleanup operation. Store the backup separately from the working storage.

  8. Verify Oak Run compatibility: Use an Oak Run version matching the deployed Oak core version (for example, Oak 1.22.20 requires oak-run-1.22.20.jar). Do not use older Oak Run versions for compaction. Refer to Oak documentation for details.

  9. After sufficient capacity is available, run or allow revision cleanup to proceed.

  10. Verify logs for successful compaction and space reclamation, such as:

    • “compaction succeeded”
    • “cleanup completed”
    • “space reclaimed greater than 0 B”
    • “RevisionCleanupTask completing successfully”
  11. If online cleanup still cannot recover space, perform a controlled offline full compaction with AEM stopped, using the matching Oak Run version and Adobe-supported procedures.

  12. After recovery, investigate the cause of repository growth by reviewing checkpoint retention, repository write activity, workflows/imports, indexing activity, and maintenance-task history.

recommendation-more-help
experience-cloud-kcs-help-kbarticles