Understanding thread Health through Javacore

Javacore is a snapshot of the current JVM's running state. By analyzing Javacore, you can understand the current state of an application running in the JVM, such as whether the card is at a certain point, or if it is running too long on some code.

Basic content of Javacore

Javacore, also known as "Threaddump" or "Javadump", is a diagnostic feature provided by Java that provides a readable snapshot of the current running JVM thread usage. That is, at a particular point in time, which threads in the JVM are running, which class each thread executes to, and which method.

An application will automatically trigger the generation of Javacore if an unrecoverable error or memory leak occurs. In order to diagnose the performance problem, we also actively trigger the generation of Javacore. In AIX, Linux, and Solaris environments, we typically use kill-3 <PID> to generate Javacore for that process.

The Javacore on the IBM Jvm,aix platform will be written to Javacore.<date>.<time>.<pid>.<sequence>.txt. For Oracle Jvm,javacore is attached to native_stdout.txt.

Although the format of the Javacore produced by different versions of the JVM is slightly different, it basically contains the following content:

TITLE Information Block: describes the cause, time, and path of the file that the Javacore produces. The most common are the following three types:

1. User:sigquit Signal
2. GPF: Program General protection error causes system crash
3. SYSTHROW:JVM thrown inside an exception

Gpinfo Information Block: GPF (General protection Error) information

Envinfo Information BLOCK: environment and JVM parameters at System runtime

Meminfo Information blocks: memory usage and garbage collection

LOCKS Information blocks: user monitor and System Monitor (monitor) scenarios

THREADS Information BLOCK: state information and execution stack for all Java threads

CLASSES Information BLOCK: class loading information

Threads in Javacore can be divided into the following states:

• Deadlock (Deadlock) "Focus on": generally refers to multiple thread calls, into the mutual resource occupation, resulting in waiting for the situation can not be released.
• In-Progress (Runnable) "Focus": generally refers to the thread is in the execution state, the thread consumes resources, is processing a request, it is possible to operate on a file, it is possible to convert data types and so on.
• Wait for resources (waiting on condition) "Focus on": Wait for the resource, if the stack information is explicitly applied code, it proves that the thread is waiting for a resource, usually a large number of read a resource, and the resource takes a resource lock, the thread enters the waiting state. Or, waiting for other threads to execute, and so on.
• Wait for the monitor to check the resource (Waiting on monitor)
• Pause (Suspended)
• Object Waiting (Object.wait ())
• Blocking (Blocked) "Focus": refers to the threads that are required to wait for a long time while the thread manager of a container is identified as blocked by the thread managers of the containers and can be understood as waiting for resources to time out. This is the case in the app's log, where CPU hunger is generally seen, or a thread has been executing information for a longer period of time.
• Stop (parked)

Through the analysis of Javacore data, combined with the understanding of specific application code logic, experienced engineers can view the original Javacore file directly from a text editor to analyze the current application's running state. The general beginner needs some tools for more intuitive analysis. Graphical analysis tool Tmda

　　

WC Thread Execution Stack analysis

Regardless of whether or not the Tmda tool is used for analysis, the analysis of Javacore will eventually be implemented on the execution stack of the specific thread. If you are unfamiliar with the code for your application, you may find it impossible to look at a long execution stack. This section describes common code and corresponding function modules for the WC thread execution stack. Beginners can speculate on the current running state of a thread based on these examples. It is important to note that the different versions of the WC, the exact code of the same function module may change, and the user-defined code is much more diverse. This section provides only some examples based on the code of the WC FEP7 version, the reader needs to be flexible in accordance with the actual code of the system he is dealing with.

Usually there are hundreds of threads in a Javacore, and these threads do not have the same status. Some threads are "ingress threads" that are run by the system, while others are simply worker threads that are derived from those threads. So be sure to grab these main threads during the Javacore analysis.

The core of the WC is a Web application, so most of the WC's Javacore are portals to the application server's Web container. The JVM used to handle the front-end store or the backend management side is basically similar, but the JVM that runs the scheduled task specifically differs. 2 is shown as a generic invocation structure with a WEB container as an ingress thread:

Starting with the Web container portal, it is generally entered into the Servlet execution. If there is a cache and hit, it will enter the relevant code of DynaCache. If there is no cache or cache misses, if it is a JSP page, the JSP code will be executed, otherwise the corresponding logic will be executed. During code logic processing, it is often possible to access the database (either through a DSL or EJB) or SOLR search (through BOD or REST), as well as access to the External System integration interface (via HTTP synchronous calls or message queues). If the database server/Search server/System integration Server is distributed across other nodes, these calls will eventually be converted to network access.

The following is an example of an execution stack that is working on a database request in a JSP page, 3, 4, 5:

Figure 3. Stack instances (1)

Figure 4. Stack instances (2)

bottom-up key code :

2. WEB Container Processing Request: Package name/class name. Method Name: Com.ibm.ws.webcontainer/webcontainer.handlerequest
4. Runtimefilter:package name/class name. Method Name: Com.ibm.commerce.webcontroller/runtimeservletfilter.dofilteraction
6. Servlet Processing: Package name/class name. Method Name: Com.ibm.commerce.struts/ecactionservlet.doget or package name/class name. Method Name: Com.ibm.commerce.struts/ecactionservlet.dopost
8. JSP processing :Package Name/class name. Method Name: Com.ibm._jsp/_ (JSP file name). _jspservice
10. Command Execution (the legend is BOD command): Package name/class name. Method Name: Com.ibm.commerce.*/abstract (*) Cmdimpl.performexecute
12. Dsl:package name/class name. Method Name: com.ibm.commerce.foundation.server.services.dataaccess/abstractdataservicefacade.*
14. Jdbc:

    Query:P ackage name/class name. Method Name: Com.ibm.ws.rsadapter.jdbc/wsjdbcpreparedstatement.executequery or package name/class name. Method Name: Com.ibm.ws.rsadapter.cci/wsresourceadapterbase.executequery Update: Package name/class name. Method Name: Com.ibm.ws.rsadapter.jdbc/wsjdbcpreparedstatement.executeupdate or package name/class name. Method Name: Com.ibm.ws.rsadapter.cci/wsresourceadapterbase.executeupdate

16. Database Driver code: DB2 Package Name: com.ibm.db2.* or Oracle package name/class name. Method Name: Oracle.jdbc.driver/oraclepreparedstatement.executeinternal
18. External network access :

    (Network Read) package name/class name. Method Name: java.net/socketinputstream.socketread0 (Network write) the package name/class name. Method Name: Java.net/socketoutputstream.socketwrite0

Other common examples of execution stacks are:

• Idle ( Idle )

  Such a stack indicates that the current thread is idle (for the Web container, that is, the current thread is not receiving a Web request). The call stack is: Package name/class name. Method Name: Java.lang/object/wait ... Package name/class name. Method Name: Com.ibm.ws.util/threadpool$worker. Run or package name/class name. Method Name: Com.ibm.io.async/asynclibrary. Aio_getioev* ... Package name/class name. Method Name: Com.ibm.ws.util/threadpool$worker. Run

• Transaction (Transaction) processing

  Commit (Commit) the package name/class name. Method Name: Com.ibm.commerce.server/transactionmanager.commit rollback (Rollback) package name/class name. Method Name: Com.ibm.commerce.server/transactionmanager.rollback

• Cache (DynaCache) read

  Package name/class name. Method Name: Com.ibm.ws.cache/cache.getentry (or Getcacheentry) WXS-based cache is the package name/class name. Method Name: Com.ibm.ws.objectgrid.dynacache/remotecorecacheimpl.get

• Multiclick Processing: Package name/class name. Method Name: Com.ibm.commerce.webcontroller.doubleclick/multiclickrequesthandler. waitForResponse
• Message processing (MQ): Package name/class name. Method Name: Com.ibm.mq.jmqi.remote.internal/remotercvthread.run
• Search (SOLR) Processing: Package name/class name. Method Name: Org.apache.solr.client.solrj/solrserver.query
• REST Processing: Package name/class name. Method Name: Com.ibm.commerce.foundation.internal.client.util/resthandler.execute

-----Organized from: http://www.kankanews.com/ICkengine/archives/156577.shtml

Understanding thread Health through Javacore