Linux operations-aware performance monitoring and debugging tools

Linux operations-aware performance monitoring and debugging tools 1 Nagios

Nagios is an open source monitoring solution, and I think he can monitor everything, and can look at my previous article: NAGIOS

2 PS #用来查看程序的运行情况

Ps-ef |grep svr.py|grep-v grep |awk ' {print $} ' #查看svr. PY process PID Number
14554

3 free# View the system's physical (RAM) and swapped memory information
total used free shared buffers cached
swap:2053 1965
total:5250 3035 2215
4 top# Displays information about all running processes, Windows-like explorer detailed see: research on top command
5 Pmap #显示一个给定的进程的内存映射

14443:bash
start SIZE RSS PSS DIRTY SWAP PERM mapping
08048000 580K 476K 74K 0K 0K r-xp/bin/b ASH
080d9000 4K 4K 4K 4K 0K R--p/bin/bash
080da000 8K 8K 8K 8K 0K rw-p/bin/bash
b744c000 8K 8K 8K 8K 0K rw-p [anon]
b744e000 252K 32K 6K 0K 0K r--p/usr/lib/locale/zh_cn.utf8/lc_ctype
b748d000 1344K 24K 2K 0K 0K r--p/usr/lib/locale/zh_cn.utf8/lc_collate

6 nestat# display Various network related information, such as network connection, routing table, interface statistics, etc., this is very common, such as determine whether the port of a program is listening

[Email protected]:~> sudo/bin/netstat-ap |grep ssh #-a means all ports-p display PID
Root ' s password:
TCP 0 0 linux-b8lh.site:59073 mta1:61002 established29323/ssh
TCP 0 0 linux-b8lh.site:35252 10.14.22.26:61300 established11986/ssh
TCP 0 0 linux-b8lh.site:57146 mta1:61002 established9138/ssh
TCP 0 0 linux-b8lh.site:32997 10.14.22.26:61300 established15122/ssh
TCP 0 0 linux-b8lh.site:45807 10.14.22.26:61300 established29240/ssh
TCP 0 0 linux-b8lh.site:35865 mta1:61002 established28751/ssh
UNIX 2 [ACC] STREAM LISTENING 13182 2929/gnome-keyring-/tmp/keyring-tmukpl/ssh
UNIX 3 [] STREAM CONNECTED 607118 29323/ssh
UNIX 3 [] STREAM CONNECTED 607117 29326/SSHFS
UNIX 3 [] STREAM CONNECTED 608427 29240/ssh
UNIX 3 [] STREAM CONNECTED 608426 29244/SSHFS

[Email protected]:~>sudo/bin/netstat-tunlp|grep name # The commands I use
Proto recv-q send-q Local address Foreign address State pid/programname
TCP 0 0 172.16.180.1:53 0.0.0.0:* LISTEN 5160/named
TCP 0 0 172.16.110.1:53 0.0.0.0:* LISTEN 5160/named
TCP 0 0 192.168.8.46:53 0.0.0.0:* LISTEN 5160/named
TCP 0 0 127.0.0.2:53 0.0.0.0:* LISTEN 5160/named
TCP 0 0 127.0.0.1:53 0.0.0.0:* LISTEN 5160/named
TCP 0 0 127.0.0.1:953 0.0.0.0:* LISTEN 5160/named
UDP 0 0 172.16.180.1:53 0.0.0.0:* 5160/named
UDP 0 0 172.16.110.1:53 0.0.0.0:* 5160/named
UDP 0 0 192.168.8.46:53 0.0.0.0:* 5160/named
UDP 0 0 127.0.0.2:53 0.0.0.0:* 5160/named
UDP 0 0 127.0.0.1:53 0.0.0.0:*

Kernel IP Routing table
destination Gateway genmask Flags MSS Window irtt iface
172.16.180.0 * 255.255.255.0 u 0 0 0 vmnet8
172.16.110.0 * 255.255.255.0 u 0 0 0 VM NET1
192.168.8.0 * 255.255.255.0 U 0 0 0 eth3
link-local * 255.255.0.0 u 0 0 0 eth3
loopback * 255.0.0.0 U 0 0 0 lo
default 192.168.8.254 0.0.0.0 UG 0 0 0 eth3
7 Iptraf

[Email protected]:~> sudo zypper install Iptraf

Network traffic real-time monitoring tool, can monitor all traffic, IP traffic, by protocol traffic, can also set filters, etc.

Start requires root permission, a graphical tool, can grab the packet, report the network card traffic situation, network card packet number, etc., I also like it to judge some network problems

8 Htop

[Email protected]:~> sudo zypper install htop

Top of color font, support mouse click, CPU usage. Memory usage, current load conditions, etc. are displayed directly, replacing the top

9 Dstat

[Email protected]:~> sudo zypper install Dstat

----total-cpu-usage-----dsk/total-- net/total----Paging-----system--
usr sys IDL Wai HiQ siq| read writ| recv send| in Out | INTCSW
25 2 72 0 0 0| 0 40k|1328b 0 | 0 0 |2029 6175
27 2 1 0 0| 0 20k|2245b 1231b| 0 0 |2191 6545
28 1 70 0 0 0| 0 0 |1610b 0 | 0 0 |2243 6815
25 2 0 0 0| 0 104k|1224b 0 | 0 0 |2075 6610

Strace #i like it for debugging and troubleshooting executable files in a Linux environment. It shows the signal received during the process used by the system call process, which is usually not a basic operation.

#strace + Executable Program

Note: You want a specific system call to use '-e '

lsof# Uninstall Mobile storage often prompted device busy, may also mistakenly delete a file that is being opened, lsof (list openfiles) is a tool that lists the open files for the current system. The application Open File descriptor list provides a lot of information about the application itself, so the lsof tool is able to view this list to help with system monitoring and troubleshooting

[Email protected]:~> lsof-a-u dongwm-d txt# View files of file type txt opened by the root user process
[Email protected]:~> lsof/home# If there are any open files in the file system when uninstalling the file system, the operation will typically fail. Then through lsof you can find out which processes are using the file system you are currently uninstalling

notificat 27261 dongwm mem REG 8,7 1512 12982245/home/dongwm/.local/share/mime /MIME.CACHE
bluefish 29204 dongwm cwd DIR 8,7 12288 12320769/home/dongwm
bluefish 29204 dongwm DEL REG 8,7 12321855/home/dongwm/.local/share/gvfs-metadata/home-4be5293d.log
bluefish 29204 dongwm DEL REG 8,7 12320866/home/dongwm/.local/share/gvfs-metadata/home

Recover files: When a file in the system is accidentally deleted, as long as there is a process in the system to access the file, then we can recover the contents of the file from the/proc directory by lsof. The idea is to lsof|grep this deleted file, if the output of the item, remember the PID and the 4th column of "number + letter" of the numbers, and then, cat/proc/that pid/fd/that number > deleted files restored

w# View the user who is currently logged on to the system

uptime# View the current load command

[Email protected]:~> uptime
17:52 started 1 days 8:41, 6 users, average load: 0.42, 0.26, 0.24
vnstat# used to count network traffic

[Email protected]:~> sudo zypper install Vnstat

Vnstati-i eth0–-months–-output/dir/month.png# Monthly Traffic chart, make scheduled Tasks +html display is more OK

[Email protected]:~> sudo vnstat-create-db eth0
Root ' s password:
Error:unable to read database "/var/lib/vnstat/eth0".
Info: A New database has been created.
[Email protected]:~> vnstat--top10 #创建数据库信息, check traffic before 10

SS #socket流量状态情况, I like

[Email protected]:~> sudo/usr/sbin/ss-o State established# View the socket that established the connection
Recv-q send-q Local address:port Peer address:port
0 0 192.168.8.46:59073 60.2.235.195:61002timer: (keepalive,60min,0)
0 0 192.168.8.46:52996 112.95.243.28:http
0 0 192.168.8.46:51066 128.121.22.144:http
0 0 192.168.8.46:38547 112.95.240.222:http
0 0 192.168.8.46:49295 74.125.71.154:http
0 0 192.168.8.46:46500 10.14.22.26:61300timer: (keepalive,48min,0)

[Email protected]:~> sudo/usr/sbin/ss-l # shows all the listening sockets
Recv-q send-q Local address:port Peer address:port
0 3 172.16.180.1:domain *:*
0 3 172.16.110.1:domain *:*
0 3 192.168.8.46:domain *:*
0 3 127.0.0.2:domain *:*
0 3 127.0.0.1:domain *:*
0 127.0.0.1:ipp *:*
0:: 1:ipp:::*
0 127.0.0.1:953 *:*
0 *:mysql *:*
0:: Sunrpc:::*
0 *:SUNRPC *:*
0::: http

[Email protected]:~> sudo/usr/sbin/ss-s# Show total value statistics by type
total:719 (kernel 769)
TCP:62 (estab 5, closed, orphaned 4, synrecv 0, timewait 2/0), ports 42

Transport Total IP IPv6
* 769--
RAW 1 1 0
UDP 12 10 2
TCP 46 43 3
INET 59 54 5
FRAG 0 0 0

Linux operations-aware performance monitoring and debugging tools