The java application runs slower and finally runs out of memory or you see an error in the logs or in the console output
OutOfMemoryError: Java heap space or
OutOfMemoryError: gc overhead limit exceeded.
Such problems can have many causes.
One possible cause is that the java application, in our case, CRX / CQ was started from the command line with the default heap memory settings of Java. This means that the jvm parameter
-Xmx was not specified. CRX or CQ need at least 256 MB of heap allocated to run. If this is the problem, then starting from the command line,ensure that the heap memory settings are set. Example:
java -Xmx512m -jar *.jar
If this is not the case, then your application could be retaining too many objects without releasing them for Garbage Collection. This is called a memory leak, see here for more information. See the section below on how to analyze memory problems in java applications.
Create heap dump
Generate a heap dump automatically
To automatically create a heap dump when running out of memory, you can add the jvm parameter
-XX:+HeapDumpOnOutOfMemoryError to generate a heap dump automatically when the application throws an OutOfMemoryError. For example,
java -Xmx256m -XX:+HeapDumpOnOutOfMemoryError -jar *.jar
This generates a heap dump file (
java_...hprof) in the process’s working directory whenever the java process runs out of memory. The process can continue to run after the heap dump was generated. Usually, one heap dump file is enough to analyze the problem.
Note: if you are using
crx-quickstart/server/start script to start your CRX instance then you can add
-XX:+HeapDumpOnOutOfMemoryError to the
CQ_JVM_OPTS variable (make sure that the variable is uncommented as well). For example:
After adding this parameter and restarting the CRX instance, verify that the new jvm option is set. Run
ps -ef | grep java from the command-line. Then check whether you see
-XX:+HeapDumpOnOutOfMemoryError as a parameter of the CRX java process.
If you need to specify a different directory to generate the heap dump in due to disk space restrictions, then you can add the
-XX:HeapDumpPath=/path/to/generate/heapdump parameter to tell the jvm where to put the file.
See here for a reference of debugging related jvm parameters.
Generate a heap dump manually
To manually generate a heap dump, run this command (jmap and jps can be found in the
bin folder of your jdk home directory):
look up the pid of the java process you are generating a heap dump for.
ps -ef | grep javaor
Ctrl+Shift+Escthen go to View = Select Columns = PID (Process Identifier) or
Run the jmap command below, replace
/path/to/generate/heapdumpfile.hprof with the location you would like to generate the heap dump file, and replace
1234 with the pid that you looked up in the previous step.
jmap -dump:format=b,file=/path/to/generate/heapdumpfile.hprof 1234
You’ll first have to change the default JVM settings regarding the Dump agents, to generate the correct dumps on user signal. There are several kinds of dumps, but you generally need the full system dump to perform a thorough memory analysis. Add the following arguments:
This “user” event happening when the JVM receives the SIGQUIT (Linux, AIX®, z/OS®, and i5/OS™) or SIGBREAK (Windows) signal from the operating system.
For more information, see the vendor’s documentation here.
Warning: Heap dump files are large and can be up to the same size on disk as your max heap -Xmx jvm parameter configuration. Make sure that you have enough disk space allocated to the directory where the dump file is generated.
Analyze heap dump
A good tool to analyze heap dumps is EclipseMAT (Eclipse Memory Analyzer): http://www.eclipse.org/mat/
This tool can’t analyze IBM JVM generated dumps. For those, there are several possibilities. IBM HeapAnalyzer will do well for the heap dumps in PHD or classic format.
For a complete system dump analysis, use the IBM Support Assistant Workbench , with IBM Monitoring and Diagnostic Tools for Java - Memory Analyzer Version 1.2 installed on top.
The heap histogram is a simple measurement of the number of live objects and memory used per Java class. Unfortunately, depending on the Java installation, the required tools may not be available or may not always work. To create a heap histogram, you first need the process id of the Java process. To get it, run
ps or (if available), run:
This Java tool gets the process ids of all running Java processes. Example:
327 3332 sun.tools.jps.Jps 3313 crx-quickstart-....jar
Now run the following command:
jmap -histo 3313
The list is sorted by total memory required (shallow: excluding referenced objects). The first 20 lines of the output are the most interesting. Example output:
JVM version is 1.5.0_20-141 Iterating over heap. This may take a while... Warning: skipping invalid TLAB for thread t@62211 Warning: skipping invalid TLAB for thread t@62467 ... SizeCountClass description ------------------------------------------------------- 1059290412916byte 1028584075255* ConstMethodKlass 628317658388char 604230414928int 4995752116201* SymbolKlass 422089675255* MethodKlass 41965126969* ConstantPoolKlass 29285606969* InstanceKlassKlass 26310086066* ConstantPoolCacheKlass 2395872149742org.apache.jackrabbit.core.query.lucene.DocId$PlainDocId 14760087003java.util.HashMap$Entry 139612858172java.lang.String 107023244593java.util.HashMap$Entry 75398410036short 73546454org.apache.jackrabbit.core.query.lucene.DocId 7201927502java.lang.Class 64070413348com.day.crx.persistence.tar.index.IndexEntry ...
To help analyzing the problem, we also need to know the following information: