Monitoring and log analysis of high CPU in Linux system

When using a Linux system, consume CPU resources too high and use scripting to troubleshoot:

1, real-time monitoring, once there is a high CPU consumption process, the program starts;

2, then the process analysis, to draw the corresponding thread;

3, the corresponding thread is located in the program log document analysis, such as Websphere Middleware has a very detailed file system;

4, for the log files in the error,worning and other detailed view, but because sometimes the log file is too large, and easy to ignore some details, if using sed and awk, combined with the four expression, can effectively locate the error and not leave out any details.

this script, through a local script and a remote script, can accurately monitor and locate the log file, and analyze the file

Local script:highcpuanalysis_l.sh:

Ready to work: Define variables

###############################################################################

#The source code is created in 10.19.90.165 and 192.168.86.198

# This script was used to an analysis of data for performance, high CPU issues on Linux '

# Usage:./highcpuanalysis.sh $IP $USER

# Author:huangtao

# Email:[email Protected]

#

###############################################################################

##########################

# Define Variables #

##########################

Export user=$1;

Export ip=$2;

# #Usage:

If [$#-eq 0] | | [$#-eq 1]

Then

echo "Unable to find USER and IP."

echo "Please rerun the script as follows:./highcpuanalysis.sh USER IP"

echo "Eg:./highcpuanalysis_l.sh root 192.168.86.198"

Exit 1

Fi

# #get The remote server ' s was Application server name

Export wasappname=$ (SSH [email protected] $IP Ps-eo Pcpu,pmem,pid,user,args | sort-k 1-r | sed-n ' 2p ' |awk ' {print $NF} ‘)

# #get The remote server ' s hostname

Export remotehostname=$ (SSH [email protected] $IP hostname)

# #get The current directory

Export dir=$ (PWD)

Section 1 Step: Copy the local analysis script to the remote host:

###############################################################################

# #Copy the script:highCpuAnalysis_r.sh to target host

echo "*********************************************************************"

echo "Step 1:"

echo "Copy the highcpuanalysis_r.sh to the remote host, and"

SCP highcpuanalysis_r.sh [email protected] $IP:/tmp/

SSH [email protected] $IP cd/tmp

SSH [email protected] $IP chmod 755/tmp/highcpuanalysis_r.sh

echo "is runing on $remotehostname ($IP)."

Step 6: Copy the analysis document generated by the remote host to the local host and delete the analysis file from the remote host

###############################################################################

# #run The script, make the script run on target remote host:

SSH [email protected] $IP/tmp/highcpuanalysis_r.sh

echo "*************************************************************************"

echo "Step 6:"

echo "Copy the report and Javacore to the local Fenxi host:"

###############################################################################

# #Copy the report and Javacore to the local host then delete them:

Export dir=$ (PWD)

SCP [email protected] $IP:/tmp/highcpureport*.

SCP [email protected] $IP:/tmp/javacore*.gz.

TAR-ZXVF javacore*.gz

# #Remove all related files in Remate server

SSH [email protected] $IP rm-f/tmp/highcpu*report*

SSH [email protected] $IP rm-f/tmp/javacore*

SSH [email protected] $IP rm-f/tmp/highcpuanaly sis_r.sh

SSH [email protected] $IP rm-f/tmp/topdashh.*

echo ""

Step 7: Show the results of the analysis

echo "*********************************************************************"

echo "Step 7:"

echo "Show All information:"

echo "Remote hostname: $remotehostname ($IP)."

echo "Remote Appserver Name: $wasappname."

echo "Report and Javacore:"

Rm-f javacore*.gz

LS-RLT highcpu*report* |tail-1

Ls-rtl javacore* |tail-3

echo "*******************************end**********************************"

Remote scripting: highcpuanalysis_r.sh

Remote Scripting Preparation: Defining variables for remote hosts (mainly monitored time and frequency)

###############################################################################

#The source code is created in 10.19.90.165 and 192.168.86.198.

# This script was used to an analysis of data for performance, high CPU issues on Linux '

# Usage:./highcpuanalysis.sh

# Author:huangtao

# Email:[email Protected]

###############################################################################

##########################

# Define Variables #

##########################

# How long the top dash H data should is taken in once (second).

Top_dash_h_val=30

# How many times dash H data should is taken.

Top_dash_h_val_t=3

# How long one javacores should is taken (second).

Javacore_val=60

# How many times javacores should is taken.

Javacore_val_t=3

# #get High CPU PID

This step gets the process ID that consumes the most CPU resources

Export pid=$ (PS -eo Pcpu,pmem,pid,user,args | Sort - k 1 - R | Head  - Ten  | sed - N ' 2p '  | awk ' {print $} ')

# #get Turn PID number to hexadecimal (from 16)

Export pid16=$ (echo "obase=10; $pid "| bc

# #check The PID if was process

Export was=$ (Ps-eo Pcpu,pmem,pid,user,args | sort-k 1-r | sed-n ' 2p ' |awk ' {print $4} ')

# #get the WAS application name

Export wasappname=$ (Ps-eo Pcpu,pmem,pid,user,args | sort-k 1-r | sed-n ' 2p ' |awk ' {print $NF} ')

# #get hostname

Export hostname=$ (hostname)

Step 2: Obtain a process PID that consumes high CPU and determine whether a process is a specific program

##########################

# Get High CPU PID #

##########################

# # put the report in/tmp/highcpureport. $pid. $hostname. Out

echo "Script execude Time:" $ (date) >/tmp/highcpureport. $pid. $hostname. Out

echo ""

if [$was = Wasuser] | | [$was = Wasadmin]

Then

echo "*********************************************************************"

echo "Step 2:"

echo "The highest CPU pid is: $pid, and the process is porcess." judge whether it is was process

Else

echo "The highest CPU pid: $pid is not a was process." | Tee-a/tmp/highcpureport. $pid. $hostname. Out

echo ""

Exit 1

Fi

Sleep 1;

echo "*********************************************************************"

under a specific user name (such as wasadmin,httpd), identify the top ten processes that consume the most CPU resources :

Step 3: Analyze a thread that consumes the most CPU resources, the usage of internal threads (or sub-processes) during a specific time period, and write to the temporary file for analysis;

#########################

#                       #

# Start Collection of: #

# * Top Dash H #

#                       #

#########################

# Start The collection of top dash H data.

echo "Step 3:"

echo "Starting collection of top dash H data ..."

echo "Need $[$TOP _dash_h_val*top_dash_h_val_t] seconds to complete this step:"

top-bh-d $TOP _dash_h_val-n $TOP _dash_h_val_t-p $pid >/tmp/topdashh. $pid. $hostname. Out

#eg: top-bh-d 30-n 3-p 7031

#eg: grep-v Swap toplog.out |grep-v Task |grep-v "Cpu (s)" |grep-v "Mem:" |grep-v top| Sort-k 1-r | head-10 | Sed-n ' 2p ' |awk ' {print $} '

#echo "Analysis of the snapshot Of/tmp/topdashh. $pid. $hostname. Out can find out the hight CPU thread";

echo "collected the top dash H data."

Sleep 2;

Section Step 4: Analyze the previous steps file to find out the top 10 of threads that consume the most CPU resources for a certain period of time (if the CPU consumes too much resources is caused by was) and generate within the specified time period JAVACORE for analysis.

###########################

# Find out the Thread of most CPUs

# and Time Consumner Top 10.

###########################

# #delete The/tmp/topdashh. $pid. $hostname when completed the data Collection

############################

# Start Collection of: #

# * Javacores #

#########################

# javacores is output to the working directory of the JVM; In most cases the <profile_root>

echo "*********************************************************************"

echo "Step 4:"

echo "Starting collection of Javacores ..."

echo "Need $[$JAVACORE _val* $JAVACORE _val_t] seconds to complete this step:"

# #clear The Javacore about this PID first:

Rm-f/opt/ibm/websphere/appserver/profiles/$wasappname/javacore* $pid *

# #then generate the Javacore

Kill-3 $pid;

echo "collected the first javacore for PID $pid."

Sleep $221

Kill-3 $pid;

echo "collected the second javacore for PID $pid."

Sleep $JAVACORE _val

Kill-3 $pid;

echo "collected the third Javacore for PID $pid."

Sleep $JAVACORE _val

# #mv the Javacore to the/tmp DIR and then zip:

Rm-f/tmp/javacore*

Mv-f/opt/ibm/websphere/appserver/profiles/$wasappname/javacore* $pid */tmp/

Cd/tmp

TAR-ZCVF javacore.$ (Date +%y%m%d ".") %h%m%s). $pid. GZ javacore* $pid *

Step 5: Show the results of the analysis and save the results to a temporary file

echo "*********************************************************************"

echo "Step 5:"

echo "Print out of the analysis infomantion:"

echo "" | Tee-a/tmp/highcpureport. $pid. $hostname. Out

echo "*********the most CPU consumner top ten PROCESS:* ********************" | Tee-a/tmp/highcpureport. $pid. $hostname. Out

Ps-eo Pcpu,pmem,pid,user,args | Sort-k 1-r |head-10 | Tee-a/tmp/highcpureport. $pid. $hostname. Out

echo "*~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~*" | Tee-a/tmp/highcpureport. $pid. $hostname. Out

echo ""

echo "****the most CPUs Consumner top *threads* from process $pid:* *******" | Tee-a/tmp/highcpureport. $pid. $hostname. Out

echo "PID USER PR NI VIRT RES SHR S%cpu%MEM time+ COMMAND" | Tee-a/tmp/highcpureport. $pid. $hostname. Out

Cat/tmp/topdashh. $pid. $hostname. out|grep-v cpu|sort-k9-n-r-k1-u |head-10 | Tee-a/tmp/highcpureport. $pid. $hostname. Out

echo "*~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~*" | Tee-a/tmp/highcpureport. $pid. $hostname. Out

echo "" | Tee-a/tmp/highcpureport. $pid. $hostname. Out

echo "****the most time C Onsumner top ten *threads* from process $pid:* ******" | Tee-a/tmp/highcpureport. $pid. $hostname. Out

echo "PID USER PR NI VIRT RES SHR S%cpu%MEM time+ COMMAND" | Tee-a/tmp/highcpureport. $pid. $hostname. Out

Cat/tmp/topdashh. $pid. $hostname. Out | Grep-v cpu|sort-k11-n-r-k1-u |head-10 | Tee-a/tmp/highcpureport. $pid. $hostname. Out

echo "*~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~*" | Tee-a/tmp/highcpureport. $pid. $hostname. Out

echo ""

echo "Pleae check the Javacore and Highcpureport. $pid. $hostname. Out under current directory."

Monitoring and log analysis of high CPU in Linux system