Common commands for Linux system monitoring

Brief introduction

Lists the operating system level monitoring commonly used several methods, it is recommended to collect the use

Cpu

The top command can be used to monitor the overall load of the system, including CPU, memory usage, and real-time display of the resource usage of each process in the system.

Output sample

top-19:37:41 up 192 days, 9:14, 1 user, load average:0.39, 0.28, 0.27tasks:126 Total, 1 running, sleeping,    0 stopped, 0 zombiecpu (s): 0.1%us, 0.1%sy, 0.0%ni, 99.8%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%stmem:32113m Total,  11351M used, 20762M free, 266M buffersswap:4091m total, 0M used, 4091M free, 304M cached                                                                                                                                                                         PID USER PR NI VIRT RES SHR S%cpu%MEM time+ COMMAND      1 root                                                                                                                                                                             0 25280 1016 968 S 0 0.0 2:37.44 Init   2 Root 20                                                                                              0 0 0 0 S 0 0.0 0:02.40 Kthreadd                                                                           3 Root 0 0 0 0 S 0 0.                                                                                                                                                                      0 3:28.59 ksoftirqd/0 5 Root 0 0 0 0 S 0 0.0 0:00                                                                                                                                                                      . kworker/u:0 6 root RT 0 0 0 0 S 0 0.0 0:11.39 MIGR  ation/0

Indicator description

• Load average:0.06, 0.60, 0.48
  System load, which is the length of the task queue. Average in 1 minutes, 5 minutes, 15 minutes, respectively.
• Tasks
  Process summary, including run-in, sleep, stop, zombie-state grooming
• Cpu (s)

Indicators	Description
0.3% US	Percentage of CPU occupied by user space
1.0% Sy	Percentage of CPU consumed by kernel space
0.0% ni	User process priority scheduling CPU percentage
98.7% ID	Percentage of idle CPU
0.0% WA	Percentage of CPU time waiting for input and output
0.0% hi	Hardware CPU Interrupt Occupancy percentage
0.0% si	Soft interrupt Occupancy Percentage
0.0% St	Virtual machine Occupancy Percentage

• Mem
  Memory statistics, including physical memory, used memory, free memory, kernel cache

• Swap
  Swap area statistics, including total, used, idle, slow stock

• Process List

Indicators	Description
Pid	Process ID
USER	User name of the process owner
PR	Priority level
NI	Nice value. Negative values indicate high priority, positive values indicate low priority
VIRT	The total amount of virtual memory used by the process, in kilobytes. Virt=swap+res
Res	The size, in kilobytes, of the physical memory that the process used and was not swapped out. Res=code+data
Shr	Shared memory size, in kilobytes
S	Process status (d= non-interruptible sleep state, r= run, s= sleep, t= track/Stop, z= zombie process)
%cpu	Percentage of CPU used by the process
%MEM	Percentage of physical memory used by the process
time+	Total CPU time used by the process, Unit 1/100 sec
SWAP	The amount of virtual memory that the process uses, the size that is swapped out, the unit kb
CODE	The amount of physical memory the executable code occupies, in kilobytes
DATA	The amount of physical memory that is used by parts other than executable code (data segment + stack), in kilobytes
COMMAND	Command name/command line

Press the F key to choose to display different indicators, very detailed
Reference documents

Memory monitoring

Vmstat refers to virtual memory statistics (meomory Statistics), which is a common real-time system monitoring tool.

Output sample

~-> vmstat 2procs-----------memory-------------Swap-------io-----system-------CPU------r B swpd free buff   Cache si so bi bo in CS us sy ID WA St 0 0 212 21250404 272916 320572 0 0 0 2 0 0 0 0 100 0 0 0 0 212 21250652 272916 320576 0 0 0 0 535 1061 0 0 100 0 0 0 0 212 21250652 2  72916 320576 0 0 0 0 496 1062 0 0 100 0 0 2 0 212 21250256 272916 320576 0 0 0 0 751 1107 0 0 99 0 0 0 0 212 21250404 272916 320568 0 0 0 0 496 1056 0 0 100 0 0 0 0 212 212504  04 272916 320568 0 0 0 0 593 1089 0 0 100 0 0 0 0 212 21250404 272916 320568 0 0 0 56 539 1074 0 0 100 0 0 0 0 212 21250404 272916 320568 0 0 0 0 589 1137 0 0 100 0 0 0 0 212 2     1250404 272916 320564 0 0 0 0 608 1154 0 0 100 0 0 0 0 212 21250404 272916 320564 0 0 0 0 601 1156 0 0 100  0 0 1 0 212 21250404 272916 320564 0 0 0 0 611 1155 0 0 100 0 0 0 0 212 21250404 272916 3205 64 0 0 0 0 599 1151 0 0 100 0 0

Indicator description

Indicators	Description
Procs.r	Number of processes waiting for CPU time slices
procs.b	The number of processes waiting for a resource, such as waiting for IO or memory exchange
Cpu.us	User state CPU Time percentage
Cpu.sy	Percentage of CPU time in kernel state, reference value us+sy<=80%
Cpu.wa	Percentage of CPU time consumed by IO waits, reference value wa<=30
Cpu.id	Idle state CPU Time percentage
Cpu.st	Virtual Machine CPU Elapsed time percentage, the presence of a super-tick may be higher
System.in	Number of device interrupts
System.cs	Number of context Switches
Memory.swpd	Amount of memory in memory swap area (KB)
Memory.free	Amount of memory in free pages (KB)
Memory.buff	The amount of memory in buffer cache, typically used in block device reading and writing
Memory.cache	The amount of memory as the page cache, generally as the file system cache
Swap.si	The number of memory swap areas entered by memory
Swap.so	Amount of memory entered by memory swap area
Io.bi	The amount of data read from the Block device (KB/S)
Io.bo	Amount of data written to the Block device (KB/S)

Reference documents
About Linux OOM Killer
Free to view available memory

Network connection

The netstat command is used to display various network related information, such as network connection, routing table, interface status, etc.

Output sample

[email protected]:~> netstat -ano |headActive Internet connections (servers and established)Proto Recv-Q Send-Q Local Address           Foreign Address         State    tcp        0      0 0.0.0.0:4444            0.0.0.0:*               LISTEN   tcp        0      0 0.0.0.0:19999           0.0.0.0:*               LISTEN   tcp        0      0 0.0.0.0:27017           0.0.0.0:*               LISTEN   tcp        0      0 0.0.0.0:6380            0.0.0.0:*               LISTEN   tcp        0      0 0.0.0.0:111             0.0.0.0:*               LISTEN   

Common usage

• Check out the number of time_wait

• Check all socket status

netstat -nat| awk '{print awk $NF}'|sort|uniq -c|sort -n  

• Check process and number of sockets

netstat -nap| awk '{print awk $NF}'|sort|uniq -c|sort -n  

• To view the number of port connections

netstat -anp |grep 27071|wc -l

Disk usage

Iostat for monitoring disk IO throughput and Resource usage

Command

iostat -d -x -k 1 10

-D to view disk health,-K for the output of more extended fields in kilobytes

Device:         rrqm/s   wrqm/s     r/s     w/s    rkB/s    wkB/s avgrq-sz avgqu-sz   await  svctm  %utilxvda              0.01     8.32    0.49    4.94     6.82    53.08    22.06     0.11   19.68   1.76   0.95xvde              0.02   841.11   33.24  304.80   331.20  4583.79    29.08     1.67    4.93   1.15  38.76dm-0              0.00     0.00   33.26 1145.95   331.20  4583.79     8.34     0.30    2.61   0.33  38.81

Indicator description

Indicators	Description
rrqm/s	The number of read requests per second of merge (the same block has a merge mechanism)
rrqm/s	The number of write requests per second of merge (the same block has a merge mechanism)
r/s	Number of Read requests submitted per second
w/s	Number of write requests committed per second
Await	Average response time for IO requests, reference value <=10MS
rkb/s	Amount of data read per second
wkb/s	Amount of data written per second
Avgrq-sz	IO average request size (sector)
Avgqu-sz	IO Average request Queue Size
Svctm	Average IO Request Execution time
%util	Percent CPU consumption

Process monitoring

The PS command can give a snapshot of the process in the current system

• Find Process methods

ps -ef |grep nscl

• Sort by CPU occupancy view the top 10 processes

-> ps -aux --sort -pcpu |head -n 10USER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMANDiotweb    9313  1.0 14.3 6277488 4708752 ?     Sl   Nov03 687:08 ./bin/mongod -f conf/mongodb.conf --authiotweb   16957  0.3 10.1 10208428 3322260 ?    Sl   Nov01 251:37 java -jar redis-stat-0.4.14.jar 127.0.0.1:6380 3 --server=63800root      2822  0.1  0.0 133196  1288 ?        Sl   Jun07 431:05 /usr/bin/vm-agentiotweb   16942  0.1  0.0  52364 13304 ?        Ssl  Nov01  95:57 ./bin/redis-server 0.0.0.0:6380

Indicator description

Indicators	Description
USER	Process Owner
Pid	Process ID
%cpu	CPU Usage
%MEM	Memory usage
Tty	Terminal
STAT	Process status
VSZ	Virtual memory consumption
Rss	Physical memory consumption
START	Start date
Time	Startup duration
COMMAND	Bash command

Reference documents

File occupancy

Lsof (list open files) is used to list the current system open file handles, including network sockets, device handles, and so on.

Output sample

~->lsof /optCOMMAND  PID   USER   FD   TYPE DEVICE  SIZE/OFF   NODE NAMEmongod  9313 iotweb  txt    REG  252,0  36409888 761870 /opt/local/mongodb/bin/mongodmongod  9313 iotweb    4w   REG  252,0   7082638 778244 /opt/local/mongodb/log/mongodb.logmongod  9313 iotweb    8uW  REG  252,0         5 778246 /opt/local/mongodb/data/mongod.lockmongod  9313 iotweb    9uw  REG  252,0        21 778247 /opt/local/mongodb/data/WiredTiger.lockmongod  9313 iotweb   15u   REG  252,0     36864 778253 /opt/local/mongodb/data/sizeStorer.wt

Indicator description

Indicators	Description
COMMAND	The name of the process
Pid	Process ID
USER	Process Owner
Fd	File descriptor
TYPE	File types, such as Dir, Reg, etc.
DEVICE	Specify the name of the disk
SIZE	Size of File
NODE	File Index node
NAME	The name of the file

Reference documents

Common commands for Linux system monitoring