Documentation

Take thread dumps from a JVM

Last update: Mon Jun 17 2024 00:00:00 GMT+0000 (Coordinated Universal Time)

A thread dump is a list of all the Java threads that are currently active in a Java Virtual Machine (JVM). This article discusses the several ways to obtain thread dumps from a JVM within Adobe Experience Manager.

Description description

Environment

Adobe Experience Manager

Issue/Symptoms

How to obtain thread dumps from a JVM on Linux, UNIX, or Windows?

A thread dump is a list of all the Java threads that are currently active in a Java Virtual Machine (JVM).

There are several ways to take thread dumps from a JVM. It is highly recommended to take more than 1 thread dump. A good practice is to take 10 thread dumps at a regular interval (for example, one thread dump every ten seconds).

Resolution resolution

Step 1: Get the PID of your Java process

The first piece of information you will need to be able to obtain a thread dump is your Java process’s PID.

The Java JDK ships with the jps command which lists all Java process ids. You can run this command like this:

jps -l 70660 sun.tools.jps.Jps 70305

Note: In Linux and UNIX, you may have to run this command as sudo -u user jps -l, where “user” is the username of the user that the Java process is running as.

If this doesn’t work or you still cannot find your Java process, (path not set, JDK not installed, or older Java version), use

  • UNIX, Linux, and Mac OS X: ps -el | grep java
  • Windows: Press Ctrl+Shift+Esc to open the task manager and find the PID of the Java process

Step 2: Request a thread dump from the JVM

jstack

If installed/available, we recommend using the jstack tool. It prints thread dumps to the command line console.

To obtain a thread dump using jstack, run the following command:
jstack -l <pid>

You can output consecutive thread dumps to a file by using the console output redirect/append directive:
jstack -l <pid> >> threaddumps.log

Notes:

  • The jstack tool is available since JDK 1.5 (for JVM on Windows, it’s available in some versions of JDK 1.5 and JDK 1.6 only).

  • jstack works even if the -Xrs jvm parameter is enabled

  • It’s not possible to use the jstack tool from JDK 1.6 to take threaddumps from a process running on JDK 1.5.

  • In Linux and UNIX, you need to run the command as the user that owns the java process:
    sudo -u java-user jstack -l <pid>

    (< java-user> should be replaced with the id of the user that the Java process is running as)

  • In Windows, if you run jstack and get the error “Not enough storage is available to process this command” then you must run jstack as the windows SYSTEM user or the user that owns the java process. You can do this by using psexec which you can download here. To run jstack as SYSTEM user, use a command like this:

    psexec -s jstack <pid>   >> threaddumps.log

    If you are unable to install psexec on the server, then you can create a .bat file containing the command and run it using the Windows task scheduler (as a different user).

  • If the java process isn’t responding, then it can sometimes help to use the option -J-d64 (on 64 bit systems), for example:
    jstack -J-d64 -l   <pid>   >> threaddumps.log

  • If the jstack command (jstack -l <pid> >> threaddumps.log) throws the error [ 1] below, then run the command as the user that owns the java process. For example:
    sudo -u sling jstack -l <pid> >> threaddumps.log

Error attaching to process: sun.jvm.hotspot.debugger.DebuggerException: Can't attach to the process: ptrace(PTRACE_ATTACH, ..) failed for 22893: Operation not permitted

sun.jvm.hotspot.debugger.DebuggerException: sun.jvm.hotspot.debugger.DebuggerException: Can't attach to the process: ptrace(PTRACE_ATTACH, ..) failed for 22893: Operation not permitted

...

Caused by: sun.jvm.hotspot.debugger.DebuggerException: Can't attach to the process: ptrace(PTRACE_ATTACH, ..) failed for 22893: Operation not permitted

...

sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal$LinuxDebuggerLocalWorkerThread.run(LinuxDebuggerLocal.java:138)

JSTACK SCRIPT

Here’s a script (adapted from the one on eclipse.org) that will take a series of thread dumps using jstack. It also takes the thread level cpu usage using top command as well.

Just run it like this:

sudo -u <user> jstackSeries.sh
<pid> <aemserveruser> <count> <delay>

For example: sudo -u aemuser jstackSeries.sh 1234 aemserveruser 10 3

  • 1234 is the pid of the Java process
  • cq5serveruser is the Linux or UNIX user that the Java process runs as
  • 10 is how many thread dumps to take
  • 3 is the delay between each dump

Note: The top output has the native thread id in decimal format while the jstack output has the nid in hexadecimal. You can match the high cpu thread from the top output to the jstack output by converting the thread id to hexadecimal.

In addition to the script above, we also have a similar Windows Powershell script and an Adobe AEM specific script on github.

Alternative ways to obtain a Thread Dump

If the jstack tool is not available to you, then you can take thread dumps as follows:

Note: Some tools cannot take thread dumps from the JVM if the commandline parameter -Xrs is enabled. If you are having trouble taking thread dumps then please see if this option is enabled.

UNIX, Mac OS X, and Linux (JDK 1.4 or lesser version)

On UNIX, Mac OS X, and Linux, you can send a QUIT signal to the Java process to tell it to output a thread dump to standard output.

  1. Run this command to do this:
    kill -QUIT <pid>

    You may need to run this command as sudo -u user kill -QUIT <pid> where “user” is the user that the Java process is running as.

  2. If you are starting CQSE using the crx-quickstart/server/start script then your thread dumps will be output to crx-quickstart/server/logs/startup.log. If you are using a third-party application server such as JBoss, WebSphere, Tomcat, or other, see the server’s documentation to find out which file the standard output is directed to.

Windows:

JDK 1.X

  1. Download javadump.exe (attached below).
  2. Start the JVM with these three arguments (they must be in the right order):
    -XX:+UnlockDiagnosticVMOptions -XX:+LogVMOutput -XX:LogFile= C:\temp\jvmoutput.log
  3. Press Ctrl+Shift+Esc to open the Task Manager.
  4. Find the PID of the Java process.
  5. From the command line, run: javadump.exe   <pid>
  6. The thread dump will appear in the jvmoutput.log file mentioned in step 2.

JDK 1.6

Get a thread dump from jconsole tool, by using a plugin: [ 0]

Here’s how you can request a thread dump:

  1. Add the following parameter to the jvm running Communique:
    Dcom.sun.management.jmxremote
  2. Download and install JDK 1.6 (if not done yet).
  3. Download and extract the Thread Dump Analyzer utility. [ 1]
  4. Run jconsole.exe of JDK 1.6:
    jconsole.exe -pluginpath /path/to/file/tda.jar
  5. Click the Thread dumps tab.
  6. Click the Request Thread Dump link.

Note: If you are running AEM 6.x and want to observe the running threads, you can request http://<host>:<port>/system/console/status-Threads to get a thread list. However, please note that these thread dumps will not work with thread dump analysis tools such as samurai or tda.

Applies to:

All Adobe Products running in a JVM

Thread dump analysis tools

[ 0] PsExec v2.42 in Sysinternals in Microsoft documentation.

[ 1] TDA - Thread Dump Analyzer on irockel/tda on Github.com.

[ 2] Java Thread Dump Analyzer on FastThread.

[ 3] IBM Thread and Monitor Dump Analyzer for Java on IBM Support Assistant Documentation.

recommendation-more-help
3d58f420-19b5-47a0-a122-5c9dab55ec7f