Resolve segment store growth caused by Groovy Console audit trails in AEM 6.5 (Forms and other solutions)
If your AEM 6.5 on‑premise or AMS environment shows sudden disk spikes and a rapidly growing TarMK segmentstore, a high‑frequency Groovy Console job could be creating large audit trail nodes under /var/groovyconsole/audit. This scenario was observed in an AEM Forms environment, but it can occur in any AEM 6.5 TarMK setup using Groovy Console.
This article explains how to identify the offending job, safely remove its audit nodes, and reclaim disk space by running offline compaction on the segment store.
Note: This scenario involves custom Groovy Console scripts and audit trails. Groovy Console is third‑party/community tooling and not part of the standard AEM product.
Description description
Environment
- Product: Adobe Experience Manager (AEM) 6.5 (including AEM Forms)
- Version: 6.5 Deployment: On-premise or Adobe Managed Services (AMS) Persistence: TarMK with segmentstore
- Operating system: Linux or Windows
Notes:
- The described issue was observed in an AEM Forms environment but can occur in any AEM 6.5 TarMK setup using Groovy Console.
- This procedure does not apply to AEM as a Cloud Service, because users do not have file system–level access to the segmentstore.
Issue/Symptoms
In an Adobe Experience Manager (AEM) 6.5 Forms on-premise or Adobe Managed Services (AMS) production environment, there is a sudden and rapid increase in disk usage. The crx-quickstart/repository/segmentstore directory grows quickly over several days and reaches hundreds of gigabytes. Online revision cleanup runs but fails to reclaim space. No recent deployments or configuration changes explain the growth.
During analysis, the following symptoms are observed:
crx-quickstart/repository/segmentstoregrows rapidly to tens or hundreds of gigabytes.- Disk usage spikes over short periods, often during weekends or off-hours.
- Online revision cleanup runs but does not significantly reduce segmentstore size.
- There are no application deployments or configuration changes during the period of growth.
- Under
/var/groovyconsole/audit/user/<year>many audit nodes exist, created by a Groovy Console scheduled job that runs every two minutes.
Investigation shows that a custom Groovy Console job, scheduled to run every few minutes, writes large audit trail entries under a user- and year-specific path such as /var/groovyconsole/audit/user/<year>… These audit nodes cause repository bloat and drive segmentstore growth.
Resolution resolution
Step 1: Identify the Groovy Console job that generates audit trails
- Open CRXDE Lite on the affected AEM Forms instance.
- Browse to the node that stores existing Groovy Console jobs, for example under
/var/groovyconsole. - Locate existing jobs with a short-interval cron expression such as
0 0/2 * * * ?, which runs every two minutes.
For steps, refer to Using CRXDE Lite in the AEM as a Cloud Service User Guide. A typical job node contains properties similar to the following:
jobTitle = Remove Old File AttachmentscronExpression = 0 0/2 * * * ?(runs every 2 minutes)scheduledJobId = <job-id>script = <groovy-script-body>output = <summary-of-job-output>
If these jobs only generate audit logs and not business-critical content, you can safely clean up their audit nodes and adjust or remove the schedule to prevent further rapid growth. For steps, refer to Audit Log Maintenance in AEM 6.
Step 2: Clean up Groovy Console audit nodes
To reduce repository size, remove the accumulated audit nodes under /var/groovyconsole/audit/user/<year>. Use an on-demand Groovy Console script rather than a new scheduled job to avoid further growth.
Important: Use Groovy Console with caution on production systems. Always test scripts in a lower environment first, verify the target path, and run them under appropriate change management procedures. For steps, refer to Audit Log Maintenance in AEM 6 in the AEM 6.5 User Guide.
In this scenario, the growth comes from Groovy Console audit trail nodes under a user- and year-specific path, for example:
/var/groovyconsole/audit/user/<year>
This path contains only audit data for the Groovy Console jobs and is safe to remove. Adjust the year segment in the path to match your environment.
Example Groovy Console script:
import javax.jcr.Session
// Adjust this path to the correct audit root for your environment.
// Example: "/var/groovyconsole/audit/user/<year>"
def path = "/var/groovyconsole/audit/user/<year>"
Session s = session // Groovy Console injects a live JCR session
if (s.nodeExists(path)) {
s.getNode(path).remove()
s.save()
println "Removed audit nodes under " + path
} else {
println "No audit nodes to remove at " + path
}
Run the script once in production under a user account that has sufficient permissions to remove nodes under /var/groovyconsole/audit/user/<year>. In many environments this is an administrative or service user, but exact permissions depend on your internal security model.
After the script completes, verify in CRXDE Lite that the audit nodes are removed and confirm that the Groovy Console job either no longer runs or runs with a less aggressive schedule and minimal logging.
Step 3: Schedule downtime and backup for offline compaction
- Plan a maintenance window during off-business hours.
- Display a maintenance page or use existing operational procedures to block user access if required.
- Create a full backup of the instance (including the AEM installation directory and datastore) before proceeding. Offline compaction changes the repository on disk and is not easily reversible. For steps, refer to Backups in the Monitoring and maintaining your Adobe Experience Manager instance.
- Stop the AEM Forms instance cleanly.
Step 4: Run offline revision compaction on the segmentstore
For steps, refer to Revision Cleanup in the AEM 6.5 User Guide. Use an oak-run version compatible with your AEM 6.5 service pack level. Ensure at least twice the current segmentstore size is available as free disk space. Run the following command from the AEM installation directory on the server hosting the instance:
java -Xmx16g -jar oak-run-1.22.21.jar compact ./crx-quickstart/repository/segmentstore
Monitor the process until it completes successfully. Do not interrupt compaction. Doing so can corrupt the repository.
Step 5: Restart AEM and validate
- Start the AEM Forms instance after compaction finishes.
- Remove any maintenance pages or load-balancer rules used during the downtime.
- Verify that all Forms functionality works as expected (authoring, submission, processing, integrations).
- Check the size of
crx-quickstart/repository/segmentstoreand disk usage to confirm that the size has decreased to expected levels.
Prevention and best practices
- Avoid high-frequency Groovy Console jobs in production. Use scheduled jobs sparingly and only when necessary.
- Keep audit logging for Groovy Console and other tools at an appropriate level and purge audit data regularly.
- Monitor the size of
segmentstoreand disk usage. Configure alerts when usage approaches defined threshold values. - Run online revision cleanup according to Adobe recommendations and perform periodic offline compaction as needed, especially after large cleanups.
- Use built-in maintenance tasks and supported APIs where possible instead of custom scripts that generate large volumes of audit data.
Notes
- Do not manually delete files from
crx-quickstart/repository/segmentstore. Direct file deletion can corrupt the repository and lead to data loss. - If online revision cleanup does not reduce segmentstore size and segmentstore continues to grow, review recent custom jobs, scripts, and bulk operations to identify the source of write activity.
- When in doubt about repository health, use the documented Oak consistency and
checktools in a clone of the environment first, and only then apply the same steps to production.