Common performance monitoring and debugging tools in linux (with common tcpdump usage) and debugging tool tcpdump
Linux has many excellent tools to help us analyze various server performance indicators and assist in development and debugging. Below, we will only list basic commands, which are generally integrated in linux without re-installation. For more detailed commands, refer to the https://github.com/brendangregg/perf-tools
A
CPU Process ProblemsCommon tools: uptime, ps, top, mpstat, pidstat, etc.
Uptime: View the system running time and average load. Load> # of CPUs may mean CPU saturation
Ps: You can view the percentage of CPU resources occupied by a process;
Top/htop/atop: The displayed information is similar to that of ps, but top can understand CPU consumption and update and Display Based on the time specified by the user;
Mpstat: You can view the average information of all CPUs and the information of the specified CPU;
Pidstat: Useful for displaying the status and time consumption of a process.
B. Memory-relatedCommon tools: free and vmstat
Free: You can view the total memory, used, idle memory, and swap usage (the swap device is used when the system does not have enough physical memory to handle all requests, A swap device can be a file or a disk partition. However, be careful that the cost of using swap is very high. If the system does not have physical memory available, swapping will occur frequently. If the data that the swap device and program are accessing is on the same file system, it will encounter a serious IO problem, this will lead to system delays or even crashes. Note that if swap is used more frequently, the server memory is insufficient;
Vmstat: Monitors virtual memory usage, idle memory, buffer, cache, and other indicators, similar to the free tool.
C. Disk I/O ProblemsCommon tools: iostat, fio, and swapon
Iostat: You can obtain the number of data blocks read/write per second and the number of all read/write blocks. You can have a general understanding of the disk read/write performance and simulate sequential and random read/write operations on disks;
Fio: Another powerful I/O stress testing tool. The biggest feature of this tool is its ease of use. It supports many file operations and covers the File Usage we can see.
Swapon: Displays the usage of the swap device, if you have started the swap device.
D. Network I/O ProblemsCommon tools: netstat, tcpdump, route, iptarf, netperf, nicstat, ping/traceroute
Netstat: It is a very useful tool for monitoring TCP/IP networks. It can display the route table, the actual network connection, and the status information of each network interface device;
Tcpdump: Monitors TCP/IP connections and directly reads data headers at the data link layer. You can specify which data packets are monitored and which control formats are to be displayed;
Route: You can set static routes for the NIC configured by the ifconfig command. The local IP address routing table displays and modifies the entry network commands;
Iptarf: You can view the network throughput of the Local Machine to obtain the network transmission rate;
Netperf: It can simulate the network sending and receiving of servers and clients and test the network throughput;
Iperf: Similar to netperf, it simulates the network sending and receiving of servers and clients, tests the maximum TCP and UDP bandwidth performance, and provides network throughput information, as well as statistics such as vibration, packet loss rate, maximum segment, and maximum transmission unit size.
Nicstat:Monitors the status of network interfaces, such as throughput, similar to the output format of iostat.
Ping/traceroute: It is common to check whether the network is smooth.
E. Development and testingCommon tools: readelf, hexdump/xxd, od, objdump, nm, telnet/nc
Readelf: Display elf file formats in readable mode, including (target file, executable file, and shared library)
Hexdump/xxd: Print the file content in hexadecimal format
Od: Optional. Print File Content in hexadecimal format.
Objdump: Disassemble machine commands
Nm: List symbols of the target file
Telnet/nc: Test Network Connection Client
F. Tracking and debuggingCommon tools: strace, ltrace, dtrace/ftrace, and blktrace
Strace:Tracks the system call time, error information, and parameter transfer of processes.
Ltrace: Tracks the time-consuming, error information, and parameter transfer of function libraries for running processes.
Dtrace/ftrace: The synthesis of the above two tools. Dtrace is a tracing tool whichruns at the system level-this means you can trace all processes, into and out of the kernel, rather than selecting a single process to trace.
Blktrace: Block I/O event tracer
G. Comprehensive and comprehensive tools: Sar/collectl, dstat, view/proc/pid/xxx various information, sysctl,/sys various information
Appendix: common usage of tcpdump
Use root User:
Tcpdump tcp port number-X-s 0-I eth1
Tcpdump host machine ip-X-s 0-I eth1
Tcpdump host machine ip and tcp port number-X-s 0-I eth1
Instructions for use:
Monitors data packets of a specified network interface
Tcpdump-I eth1
If no Nic is specified, the default tcpdump only monitors the first network interface, which is usually eth0. In the following example, no network interface is specified.
Monitor data packets of a specified host
Print all data packets that enter or exit sundown.
Tcpdump host sundown
You can also specify an ip address, for example, intercepting all packets received and sent by all 210.27.48.1 hosts.
Tcpdump host 210.27.48.1
Print the packets that helios communicates with hot or ace.
Tcpdump host helios and \ (hot or ace \)
Intercept communication between host 210.27.48.1 and host 210.27.48.2 or 210.27.48.3
Tcpdump host 210.27.48.1 and \ (210.27.48.2 or 210.27.48.3 \)
Print the IP packet that ace communicates with any other host, but does not include the packet with helios.
Tcpdump ip host ace and not helios
To obtain an IP packet for all hosts except 210.27.48.1 and 210.27.48.2, run the following command:
Tcpdump ip host 210.27.48.1 and! 210.27.48.2
Intercept all data sent by host hostname
Tcpdump-I eth0 src host hostname
Monitor all data packets sent to host hostname
Tcpdump-I eth0 dst host hostname
Reference: https://github.com/brendangregg/perf-tools
Http://crtags.blogspot.com/2012/04/dtrace-ftrace-ltrace-strace-so-many-to.html
Https://danielmiessler.com/study/tcpdump/
