Linux server performance tracking and Common commands for server monitoring

This article will summarize some common Linux server performance tracking and Common commands for server monitoring. For more information, see.

During website performance optimization, we often need to check various Server metrics to quickly find the evil horse. In most cases, we will use monitoring software such as cacti, nagois, and zabbix, but such software is difficult to install. On a small server, we want to locate the problem as soon as possible, we still need to use some commands provided by Linux.

1. Memory check: free-m

Note: focus on the second row:-/+ buffers/cache. The first number is the actually consumed memory, and the second is the actually idle memory.

2. System Process Monitoring top
Top Command. Check the second and third lines of the output. The second line (Tasks) indicates the number of processes running by the system. This value cannot be too large, for example, more than 300, or even 400-500. This is easy to understand, generally, the services, processes, and system processes we enable are not too many, and there must be too many problems. The third line indicates the CPU usage.
Top can be sorted by M by memory usage, and P by CPU usage.

3. Process Monitoring enhanced htop
Htop provides more powerful features than top, and is recommended for installation. Supports sorting, searching, and displaying process trees.
Htop also has a powerful function, namely strace, which can be used to directly execute strace on the selected process by s.

Strace:

4. Find the evil horse (ps command)
If we analyze the system through top, htop, free, iostat, and vmstat (as described later), we will look for the system problem process. In top and htop, We can intuitively see that the process has a problem, but sometimes the two commands cannot be completely determined. We also need to use ps to confirm the worst process in the system.

Custom output format-o can define the columns we want to display. The column names are basically the same as the output header, but must be in lower case, as shown below:
Ps-A-o % mem, % cpu

Find processes that run too many instances (server monitoring is particularly useful)

We can see that there are many httpd processes (uniq may be as long as there are all rows, no repeated rows are counted, and re-count), so we can add a sort (k indicates sorting by that column ), as follows:

In this way, the httpd process can see 18 abnormal server behaviors. To differentiate users, you can add an output, for example:

ps -A -o user,comm |sort -k2| uniq -c |sort -k1 -rn

Find the process with the most memory usage

ps -eo rss,pmem,pcpu,vsize,args |  sort -k 1 -r -n

Similarly, find the process with the most CPU consumption

ps -eo rss,pmem,pcpu,vsize,args |  sort -k 2 -r -n

(The main thing is to understand sort-k and ps-o)

4. IO monitoring iostat
Iostat is also a common command for system monitoring. You can see the IO status of each disk. Other parameters are not described here. Iostat-x outputs the % util column.
Definition:
% Util: the percentage of time in one second is used for I/O operations, or the number of I/O queues in one second is not empty. That is, delta (use)/s/1000 (because the Unit of use is milliseconds)
If % util is close to 100%, it indicates that too many I/O requests are generated and the I/O system is fully loaded. This disk may have a bottleneck.
In general, % util should be smaller, the better, 10% or lower is normal, and 30% IO is busy. More than 50% of them are normal. % Idle can be used together. If the value is too small, the system is definitely faulty.

5. vmstat System Monitoring
The difference between vmstat and I/O is poor, but the focus is to monitor cpu, memory, and other system resources.
For memory monitoring, we are more concerned about swpd, free, si, and so. Generally, when the system is not busy, we can see that the value of swpd and so does not continue to be very high and is often 0. If swpd is too high, the system memory is often insufficient.
For CPU monitoring, we can view r (number of running processes), us, sy, id (idle CPU). If the number of r is greater than the number of CPU, if the CPU usage is too small, you can identify that the CPU usage is insufficient through id (CPU idle ratio) analysis.
Other parameters are not described in detail.

6. Network Traffic Monitoring iftop
Finally, let's talk about network traffic monitoring. network traffic is also very important for the system. Install iftop to monitor network browsing resources. Due to the difficulty in page scaling, I found a picture on the Internet for reference:

The left and right arrows in the middle indicate the direction of traffic.
TX: Send traffic
RX: receive traffic
TOTAL: TOTAL traffic
Cumm: total traffic from running iftop to current time
Peak: traffic peak
Rates: average traffic in the past 2 s, 10 s, and 40 s respectively

Several of the commands are useful. The display format of the switch by t is 2 rows/1 line/only shows the sent traffic/only shows the received traffic; whether the switch by T shows the total traffic of each connection; press p to switch whether port information is displayed. For other commands, press h to view the help information.

Append a command. nethogs and iftop are port and host traffic queries. However, iftop cannot query the network traffic of a specific process. Nethogs officially said nethogs = nettop, that is, to find out all processes by traffic usage, which can help us find the process with the most bandwidth usage

Nethogs is a network traffic monitoring tool on a terminal. It displays the bandwidth usage of each process, so that the network usage can be obtained more intuitively. It supports IPv4 and IPv6 protocols, local NICs, and PPP connections.

Project home: http://nethogs.sourceforge.net/

# Installation: This program is available in the official Ubuntu source.

The Code is as follows:	Copy code
Sudo apt-get install nethogs

# Use:

The Code is as follows:	Copy code
Sudo nethogs eth0

This program requires the root permission. By default, the eth0 Nic is monitored. Therefore, you can directly enter sudo nethogs. If there are multiple NICs, you must specify