Linux High CPU Analysis

1.1 Viewing CPU consumption values

Typically, this type of failure occurs when the user has a long response time, the WebLogic server is running abnormally slow, the request or operation times out, and so on. After receiving the fault notification, log on to the problem machine and execute the View process command:
Ps–ef | grep java

Here we will select the process to be viewed according to the specific alarm content:
SXYDFW 9391 9342 20:10 pts/1 01:00:22/app/wls10/jdk1.6.0_45/bin/java-server-xms1536m-xmx1536m-xx:permsize=128m-xx : maxpermsize=256m-dweblogic.name=ydfwserver2-djava.security.policy=/app/wls10/oracle/middleware/wlserver_10.3/ server/lib/weblogic.policy-dweblogic.productionmodeenabled=true-dweblogic.security.ssl.trustedcakeystore=/app/ Wls10/oracle/middleware/wlserver_10.3/server/lib/cacerts-dport=7001-dappname=sxydfw_ydfwserver2-da- dplatform.home=/app/wls10/oracle/middleware/wlserver_10.3-dwls.home=/app/wls10/oracle/middleware/wlserver_10.3 /server-dweblogic.home=/app/wls10/oracle/middleware/wlserver_10.3/server-dweblogic.management.discover=false- Dweblogic.management.server=t3://10.190.118.154:37001-dfile.encoding=gbk-dwlw.iterativedev=false- Dwlw.testconsole=false-dwlw.logerrorstoconsole=false-dweblogic.ext.dirs=/app/wls10/oracle/middleware/patch_ wls1035/profiles/default/sysext_manifest_classpath:/app/wls10/oracle/middleware/patch_ocp360/profiles/default/ Sysext_maNifest_classpath WebLogic. Server

1.2 Get Process Number
From the process information can be learned that the PID of the process is 9391, is initiated by the application account SXYDFW. Next we will look at the specific CPU usage of the process:
Perform a command to view the CPU usage of the process:

TOP-HP 9391-d 1-n 1

PID USER PR NI VIRT RES SHR S%cpu%MEM time+ COMMAND
9509 SXYDFW 0 2213m 815m 29m R 59.7 10.4 6:57.44 Java
9422 SXYDFW 0 2213m 815m 29m R 57.7 10.4 7:12.52 Java
10017 SXYDFW 0 2213m 815m 29m R 41.8 10.4 5:26.36 Java
9507 SXYDFW 0 2213m 815m 29m R 39.8 10.4 7:26.53 Java
9508 SXYDFW 0 2213m 815m 29m R 39.8 10.4 9:11.57 Java
9632 SXYDFW 0 2213m 815m 29m R 39.8 10.4 3:03.78 Java
10116 SXYDFW 0 2213m 815m 29m R 39.8 10.4 1:44.88 Java
10010 SXYDFW 0 2213m 815m 29m R 31.8 10.4 4:28.82 Java
10014 SXYDFW 0 2213m 815m 29m R 25.8 10.4 3:04.17 Java
9419 SXYDFW 0 2213m 815m 29m R 19.9 10.4 12:03.31 Java

The command can be repeated several times in intervals of 5 seconds, repeatedly gathering information to determine the consumption of threads. As can be seen from the above, in front of the CPU is occupied with high thread information. where threads 9509 and 9422 account for more than 50% of the CPU, we want to specifically analyze these two threads.

2.1 Monitoring logs

Now the WebLogic process in the production environment is started by the NOHUP standard start-stop script, so now go to the Nohup log directory of the problem server and use the TAIL–F command to monitor the log in the background.
The background log in this instance is/applog/sxydfwlog under Nohup_start_ydfwserver2_xxxx.log, so execute the command:

Tail–f Nohup_start_ydfwserver2_xxxx.log

Notify the Application support Group on the day of the teacher, ask him to use the application account in the background to perform the kill-3 process number operation, note to be executed every 15 seconds, executed 3 times.
This example takes the process number to operate:

Kill-3 9391

At this point, the thread dump information about the process appears in the monitored log nohup_start_ydfwserver2_xxxx.log, and then the two thread numbers previously obtained are converted from decimal to 16 binary. Find the corresponding content in the Threaddump information:

9509-2525:

"[ACTIVE] Executethread: ' 4 ' for queue: ' Weblogic.kernel.Default (self-tuning) '" Daemon prio=10 tid=0x00002aaab4ebb000 nid=0x2525 ru
nnable [0x00000000442a8000]
Java.lang.Thread.State:RUNNABLE
At Java.io.RandomAccessFile.writeBytes (Native Method)
At Java.io.RandomAccessFile.write (randomaccessfile.java:482)
At.com.portal.struts.action.common.UpLoadFileServlet.mergeFile (uploadfileservlet.java:224)
At Com.portal.struts.action.common.UpLoadFileServlet.doPost (uploadfileservlet.java:152)
At Javax.servlet.http.HttpServlet.service (httpservlet.java:727)
At Javax.servlet.http.HttpServlet.service (httpservlet.java:820)
At Weblogic.servlet.internal.stubsecurityhelper$servletserviceaction.run (stubsecurityhelper.java:227)
At Weblogic.servlet.internal.StubSecurityHelper.invokeServlet (stubsecurityhelper.java:125)
At Weblogic.servlet.internal.ServletStubImpl.execute (servletstubimpl.java:300)
At Weblogic.servlet.internal.TailFilter.doFilter (tailfilter.java:26)
At Weblogic.servlet.internal.FilterChainImpl.doFilter (filterchainimpl.java:56)
At Com.portal.filter.EncodeFilter.doFilter (encodefilter.java:48)
At Weblogic.servlet.internal.FilterChainImpl.doFilter (filterchainimpl.java:56)
At Weblogic.servlet.internal.webappservletcontext$servletinvocationaction.wraprun (WebAppServletContext.java : 3715)
At Weblogic.servlet.internal.webappservletcontext$servletinvocationaction.run (webappservletcontext.java:3681)
At Weblogic.security.acl.internal.AuthenticatedSubject.doAs (authenticatedsubject.java:321)
At Weblogic.security.service.SecurityManager.runAs (securitymanager.java:120)
At Weblogic.servlet.internal.WebAppServletContext.securedExecute (webappservletcontext.java:2277)
At Weblogic.servlet.internal.WebAppServletContext.execute (webappservletcontext.java:2183)
At Weblogic.servlet.internal.ServletRequestImpl.run (servletrequestimpl.java:1454)
At Weblogic.work.ExecuteThread.execute (executethread.java:209)
At Weblogic.work.ExecuteThread.run (executethread.java:178)

9422-> 24ce:
"[ACTIVE] Executethread: ' 1 ' for queue: ' Weblogic.kernel.Default (self-tuning) '" Daemon prio=10 tid=0x00002aaab4bfb800 Nid=0x24ce ru
nnable [0x0000000042389000]
Java.lang.Thread.State:RUNNABLE
At Java.io.RandomAccessFile.writeBytes (Native Method)
At Java.io.RandomAccessFile.write (randomaccessfile.java:482)
At Com.portal.struts.action.common.UpLoadFileServlet.mergeFile (uploadfileservlet.java:224)
At Com.portal.struts.action.common.UpLoadFileServlet.doPost (uploadfileservlet.java:152)
At Javax.servlet.http.HttpServlet.service (httpservlet.java:727)
At Javax.servlet.http.HttpServlet.service (httpservlet.java:820)
At Weblogic.servlet.internal.stubsecurityhelper$servletserviceaction.run (stubsecurityhelper.java:227)
At Weblogic.servlet.internal.StubSecurityHelper.invokeServlet (stubsecurityhelper.java:125)
At Weblogic.servlet.internal.ServletStubImpl.execute (servletstubimpl.java:300)
At Weblogic.servlet.internal.TailFilter.doFilter (tailfilter.java:26)
At Weblogic.servlet.internal.FilterChainImpl.doFilter (filterchainimpl.java:56)
At Com.portal.filter.EncodeFilter.doFilter (encodefilter.java:48)
At Weblogic.servlet.internal.FilterChainImpl.doFilter (filterchainimpl.java:56)
At Weblogic.servlet.internal.webappservletcontext$servletinvocationaction.wraprun (WebAppServletContext.java : 3715)
At Weblogic.servlet.internal.webappservletcontext$servletinvocationaction.run (webappservletcontext.java:3681)
At Weblogic.security.acl.internal.AuthenticatedSubject.doAs (authenticatedsubject.java:321)
At Weblogic.security.service.SecurityManager.runAs (securitymanager.java:120)
At Weblogic.servlet.internal.WebAppServletContext.securedExecute (webappservletcontext.java:2277)
At Weblogic.servlet.internal.WebAppServletContext.execute (webappservletcontext.java:2183)
At Weblogic.servlet.internal.ServletRequestImpl.run (servletrequestimpl.java:1454)
At Weblogic.work.ExecuteThread.execute (executethread.java:209)
At Weblogic.work.ExecuteThread.run (executethread.java:178)

Analyze the above information to check what causes the high CPU utilization. In this case, the execution of a function caused a failure, it is necessary to contact the development team teachers to focus on troubleshooting the highlighted part of the method call situation. If you have a problem with WebLogic, you can contact BEA customer support.

Linux High CPU Analysis