AEM Thread Dump Analysis

Download and install IBM Thread Analyzer (we’ll call it IBM TDA for short).

Capture thread dumps from an AEM instance experiencing performance issues.

Open the thread dumps in IBM TDA.

To view the details of a thread dump, select the file in the listing, then click the Thread Detail button.

Sort by Stack Depth with longest stacks on top.

Review the threads with stack depth of 10 lines or longer.  Those are usually the threads of most interest.

Take notes on threads that are of interest.

Sort by thread State.

Scroll down to the Runnable threads. Runnable threads are the ones that were actively taking up CPU time when the thread dump was taken.

Note: When reviewing the Runnable threads, you can ignore threads listed in the  Threads that can be ignored  section at the bottom of this page.

Find runnable threads that are part of the application, for example, background job threads or request threads (request threads have names like this -  127.0.0.1 [ 1347028187737] GET /content/sites/global/en/sitemap.static-delivery.httpd.html HTTP/1.1).

Once you find them then click on them one by one.

For each request thread, you can find out when the user’s browser made the request to the server by looking at the timestamp in the thread name.

For example, in the thread name above, the timestamp (in millisecond unix epoch format) is  1347028187737.

We can convert that epoch number to a date / time using www.epochconverter.com.

Each thread dump shows the date and time when it was taken.

You can take the difference in time between the request time and the thread dump time to see how long a request has been active for.

After reviewing the request threads, scroll through the other Runnable threads.

Once you have found a Runnable thread of interest, then look at the middle panel, Waiting threads.

Threads listed there are waiting for the selected thread to release a monitor.

If you do not see any waiting threads, then your selected thread could still be the owner of a Lock (see implementing classes of Lock for details).

For example, with a ReentrantReadWriteLock you can’t tell which thread is the lock holder as locks implement multiple monitors internally.

So you might have to look at the source code to match it up with a thread that could be the lock holder.

If the thread had a lock or monitor that many other threads were waiting on, then go through the rest of the thread dumps to see if you can find other threads that have the same issue.

See if the same thread still exists in the other dumps (in IBM TDA you can select multiple thread dumps and click the Compare Threads button to view a thread’s state across multiple thread dumps.

See the Collector Service in the screenshot below:

In this view you can see the thread across multiple thread dumps to see if it is a long running thread.

Basically if the thread is in the  Runnable state across multiple dumps and has a long stack, then that usually means it is a long running thread.

If you didn’t find much looking at the Runnable threads, then go back to the thread listing, select a thread dump, then click on Monitor Detail button on the top panel.

IBM TDA will open a window showing a tree view of monitor owning threads and their waiting threads.

Note: It might display some thread pool threads, like the servlet engine thread pool monitor, idle threads could be ignored.

You can usually tell a thread is an idle thread pool thread because most of the time they only have 10 stack lines or less.

If you captured top -H -b -n1 -p <javapid> output in addition to thread dumps, then you can cross-reference the thread-level CPU utilization.

Open the top output and get the process id of the threads that are utilizing the CPU.

Convert the process id to hexadecimal, then search for that hexadecimal value in the corresponding thread dump file.

The id should match the nid of one of the threads.

If the matching thread utilizing the most CPU is the VM Thread or any GC threads then you might have a memory issue.

Repeat the same exercise for more thread dumps and top output, and if there is a pattern of these threads taking CPU time, you have a memory issue.

If you have confirmed the memory issue, then capture a heap dump next time the problem occurs.

See this Analyze Memory Problems article for more details on capturing and analyzing heap dumps.