Analyze AEM Java thread dumps using IBM Thread Analyzer tool.
5. Sort by “Stack Depth” with longest stacks on top.
6. Review the threads with stack depth of 10 lines or longer. Those are usually the threads of most interest. Take notes on threads that are of interest.
7. Sort by thread “State”
8. Scroll down to the “Runnable” threads. Runnable threads are the ones that were actively taking up CPU time when the thread dump was taken.
Note: When reviewing the “Runnable” threads, you can ignore threads listed in the Threads that can be ignored section on the bottom of this page.
9. Find runnable threads that are part of the application, for example, background job threads or request threads (request threads have names like this 127.0.0.1 1347028187737 GET /content/sites/global/en/sitemap.static-delivery.httpd.html HTTP/1.1). Once you find them then click on them one by one.
10. For each request thread, you can find out when the user’s browser made the request to the server by looking at the timestamp in the thread name. For example, in the thread name above, the timestamp (in millisecond unix epoch format) is 1347028187737. We can convert that epoch number to a date / time using www.epochconverter.com. Each thread dump shows the date and time when it was taken. You can take the difference in time between the request time and the thread dump time to see how long a request has been active for.
11. After reviewing the request threads, scroll through the other “Runnable” threads. Once you have found a “Runnable” thread of interest, then look at the middle panel, “Waiting threads”. Threads listed there are waiting for the selected thread to release a monitor. If you do not see any waiting threads, then your selected thread could still be the owner of a Lock (see implementing classes of Lock for details). For example, with a ReentrantReadWriteLock you cannot tell which thread is the lock holder as locks implement multiple monitors internally. So you might have to look at the source code to match it up with a thread that could be the lock holder.
12. If the thread had a lock or monitor that many other threads were waiting on then go through the rest of the thread dumps to see if you can find other threads that have the same issue. See if the same thread still exists in the other dumps (in IBM TDA you can select multiple thread dumps and click the “Compare Threads” button* to view a thread’s state across multiple thread dumps.
13. See the Collector Service in the screenshot below:
14. In this view you can see the thread across multiple thread dumps to see if it is a long running thread. Basically if the thread is in Runnable state across multiple dumps and has a long stack then that usually means it is a long running thread.
15. If you didn’t find much looking at the runnable threads then go back to the thread listing, select a thread dump then click on “Monitor Detail” button* on the top panel. IBM TDA will open a window showing a tree view of monitor owning threads and their waiting threads. Note: It might display some thread pool threads like the servlet engine thread pool monitor, idle threads could be ignored. You can usually tell a thread is an idle thread pool thread because most of the time they only have 10 stack lines or less.
Thread level CPU utilization (Linux platform only):
Threads that can be ignored: