Full coverage of Linux monitoring commands

1.1 top1.1.1 Command Description

The TOP command monitors the system's operational status in real time and can be sorted by CPU, memory, and execution time

1.1.2 Parameter Description

• command-line Startup parameters:
• Usage: TOP-HV | -bcisshm-d delay-n iterations [-u user |-u user]-P pid [, PID ...]
• -B: Batch mode runs. Typically used to send the results of the top output to another program or to a file
• -C: Displays the command line that performs the task
• -D: Set delay time
• -H: Help
• -H: Displays the thread. When this setting is turned on, all threads generated by the process are displayed
• -I: Show idle Processes
• -N: Number of executions. Commonly used with-B
• -U: Monitor specified user-related processes
• -U: Monitor specified user-related processes
• -P: monitors the specified process. When multiple processes are monitored, the process IDs are separated by commas. This option can only be used at the command line
• -S: Safe mode operation
• -S: Cumulative time mode
• -V: Displays the top version and then exits.
• -M: Automatic display of memory units (k/m/g)

1. Global Commands

• Enter, Space: Refresh display information
• ?, h: Help
• =: Remove the limit displayed for all tasks
• A: Alternate display mode toggle
• B: Bold Display Toggle
• D, S: Change the interface refresh time interval
• G: Select a different window/field group
• I:irix or Solaris mode switching
• u, u: monitor specified user-related processes
• K: End Process
• Q: Exit Top
• R: Re-set the nice value of the process
• W: Store Current settings
• Z: Change the color template

2. Summary Area Command

• L: Average load and system run time display switch
• M: Memory and swap space usage display switch
• T: Current task and CPU status display switch

1: summary displays CPU status or shows each CPU state separately

1. Task Area Commands

• Appearance style
• B: bold/Inverse color to display highlighted rows/columns. Control the display style for x and y interactive commands
• X: Highlight sorted columns
• Y: Highlight a running task
• Z: Color/black and white display.
• Show content
• C: Command line or process name for task execution
• F, O: Add and remove process information fields and adjust process information field display order
• H: Show Threads
• S: Time Accumulation Mode
• U: Monitor specified user-related processes
• The number of tasks displayed
• I: Show the Idle Process
• N or #: Sets the maximum number of tasks to display
• Task sort (shift+f )
• M: Sort by Memory utilization
• N: Sort by PID
• P: Sort by CPU usage
• T: Sort by time+
• <: Sort by adjacent columns to the left of the current sort field
• >: Sort by adjacent columns to the right of the current sort field
• F or O: select sort field
• R: Reverse Sort

1.1.3 Result Description

1.2 Free

1.2.1 Command description

The free command is the most common command for monitoring system memory

1.2.2. Parameter description

-M: View memory usage in m (default is KB)

-B: View memory usage in bytes

-S: Can not easily monitor memory usage for a specified period of time

1.2.3 Result Description

• Total: The amount of physical memory.
• Used: How large is used.
• Free: How much is available.
• Shared: The total amount of memory shared by multiple processes.
• Buffers/cached: The size of the disk cache.

1.3 vmstat1.1.1 Command Description

You can monitor the operating system's process state, memory, virtual memory, disk IO, context, and CPU information.

1.1.2 Parameter Description

Vmstat [-A] [-n] [-s unit] [delay [count]]

• -A: Show active and inactive memory
• -M: Display Slabinfo
• -N: Displays the field names only once at the beginning.
• -S: Displays memory-related statistics and the number of system activities.
• Delay: Refresh time interval. If not specified, only one result is displayed.
• Count: Number of refreshes. If you do not specify the number of refreshes, but the refresh interval is specified, the number of refreshes is infinite.
• -D: Displays statistics about each disk.
• -S: Displayed using the specified units. The parameters are K, K, M, M, respectively, representing 1000, 1024, 1000000, 1048576 bytes (byte). The default unit is K (1024x768 bytes)
• -V: Displays Vmstat version information.
• -P: Display specified disk partition statistics
• -D: Show disk overall information

1.1.3 Result Description

Procs

R: Number of processes waiting to be executed, that is, the number of processes running and waiting for CPU time slices

B: Number of queued processes, that is, the number of processes waiting for a resource

Memory

Swap: Virtual memory, the amount of memory to switch to virtual memory

Free: idle physical memory size

Buff: Buffer size

Cache: Buffer Size

Swap

Si: The disk is written to virtual memory, which is the size of the virtual memory entered by the memory.

So: virtual memory is written to disk, which is the size of virtual memory entering the disk.

Io

Bi: The total amount of data read by the block device, read disk

Bo: The total amount of data written by the block device, write disk

System

In: Number of device interrupts per second

Cs: Number of context switches per second

Cpu

Us: User process consumes CPU percent

Sy: Kernel process consumes CPU percent

Id:cpu percentage of time in idle state

Wa:io the percentage of time that the CPU is waiting to occupy

1.4 iostat1.4.1 Command Description

Iostat is the system disk IO operation monitoring, its output mainly shows the disk read and write operations statistics. At the same time, the use of the CPU

1.4.2 Parameter Description

Iostat [-C |-d] [-K |-m] [-t] [-v] [-x] [device [...] | ALL] [-P [Device | all] [interval [count]]
The options and the parameters have the following meanings:

• -C: Displays only CPU statistics. Mutually exclusive with the-D option.
• -D: Displays only disk statistics. Mutually exclusive with-c option.
• -K: Displays the number of disk requests per second, in units of K, the default unit block.
• -p:device | All
  Mutually exclusive with the-X option to display statistics for block devices and system partitions. You can also specify a device name after-p, such as:
  # iostat-p had

or Show All devices
# iostat-p All
-T: When the data is output, the time to compile the data is printed.
-V: Prints the version number and help information.
The-X device output specifies the disk device name to be counted, and defaults to all disk devices.

-Interval: Refers to two times of statistical interval

-Count: The number of times that are counted according to the time interval specified by interval

1.4.3 Result Description

Simple application of Iostat

Iostat Disk Monitoring

• RRQM/S: The number of read operations per second for the merge, Delta (rmerge)/S.
• WRQM/S: The number of write operations per second for the merge, Delta (wmerge)/S.
• R/S: Number of Read I/O devices completed per second, Delta (Rio)/S.
• W/S: Number of write I/O devices completed per second, Delta (WIO)/S.
• RSEC/S: Number of Read sectors per second, Delta (rsect)/S.
• WSEC/S: Number of Write sectors per second, Delta (wsect)/s
• rkb/s: Reads K bytes per second, is half of rsect/s, because the size of each sector is 512 bytes.
• wkb/s: Write K bytes per second, half of wsect/s
• Avgrq-sz: The average data size (sector) for each device I/O operation, Delta (rsect+wsect)/delta (rio+wio).
• Avgqu-sz: The average I/O queue length, which is Delta (AVEQ)/s/1000 (because Aveq is in milliseconds).
• Await: The average wait time (in milliseconds) for each device I/O operation, Delta (ruse+wuse)/delta (rio+wio).
• SVCTM: The average service time Per device I/O operation (in milliseconds), which is Delta (use)/delta (Rio+wio).
• %util: How much time in a second is spent on I/O operations, or how many times in a second I/O queues are non-empty,
  That is, the delta (use)/s/1000 (because the use is in milliseconds).

Iostat CPU Monitoring

• %USR: Percentage of CPU time consumed by the user process.
• %nice: The percentage of CPU time that is consumed by running the normal process.
• %system: The percentage of CPU time that the system process consumes.
• %IOWAIT:I/O the percentage of CPU time that the wait is taken.
• %steal: In a memory-intensive environment, Pagein enforces steal operations on different pages.
• The percentage of time%idle:cpu idle state.

1.5 mpstat1.5.1 Command Description

Mpstat can monitor some of the CPU's statistics, in the multi-core CPU system can not only view all the CPU's average status information, but also can view the specific CPU information

1.5.2 Parameter Description

Mpstat [-P {| All}] [internal [count]]

Parameters:

(1)-P {| All}: Indicates which CPU to monitor, and value in [0,cpu number-1];

(2) Internal: The interval between the adjacent two samples;

(3) Count: Number of samples, count can only be used with delay;

Note: When there are no parameters, Mpstat displays the average of all information after the system is started. When there is interval, the first line of information is the average information since the system started. Starting with the second line, the output is the average information for the previous interval time period.

1.5.3 Result Description

(1) User: In internal time period, the CPU time (%), does not contain the Nice value is negative process, the value is (usr/total) *100;

(2) Nice: In the internal time period, the nice value is the CPU time of the negative process (%), the value is (nice/total) *100;

(3) System: In the internal time period, the core time (%), the value is (system/total) *100;

(4) Iowait: In the internal time period, the hard disk IO wait time (%), the value is (iowait/total) *100;

(5) IRQ: In the internal time period, the hard Interrupt time (%), the value is (irq/total) *100;

(6) Soft: In the internal time period, the soft interrupt time (%), the value is (softirq/total) *100;

(7) Idle: During the internal time period, the CPU drops idle time (%) for any reason other than waiting for the disk IO operation, the value is (idle/total) *100;

(8) INTR/S: In the internal time period, the number of interrupts received by the CPU per second, the value is (intr/total) *100;

1.6 sar1.6.1 Command Description

The SAR command can obtain the full name of the CPU, run, disk IO, virtual memory, memory, network and other information.

1.6.2 parameter Description

Common formats for the SAR command line:
SAR [Options] [-A] [-o file] t [n]
In the command line, the N and t two parameters are combined to define the sampling interval and the number of times, T is the sampling interval, is a required parameter, n is the number of samples, is optional, the default value is 1,-o file means the command results in binary format in the file, where file is not the keyword, is the file name. Options is a command-line option, there are a number of choices for the SAR command, and only the common options are listed below:

• -A: The sum of all reports.
• -U:CPU Utilization
• -V: Process, node, file, and lock table state.
• -P: This is like specifying CPU usage information in the current system.
• -D: Hard drive usage report.
• -r: Displays the usage of system memory.
• -N: Displays network operational status. Parameters can be followed with Dev, Edev, sock, and full. DEV Displays the network interface information, Edev displays the statistics of the network error, sock displays the socket information, full displays all the information of the first three parameters.
• -Q: Displays the size of the running queue, which is the same as the average load at the time of the system
• -B: Memory Paging situation
• -r: Displays the activity of the process over the sample time.
• -G: Serial I/O condition.
• -B: Buffer usage.
• -A: File read and write.
• -C: System call condition.
• -R: The activity of the process.
• -Y: Terminal equipment activity situation.
• -W: System Exchange activity.

1.6.3 Result Description

CPU Resource Monitoring

Cpu:all represents the average of the statistics for all CPUs.

%user: Displays the percentage of total CPU time that is running at the user level (application).

%nice: The percentage of total CPU time that is displayed at the user level for the nice operation.

%system: The percentage of total CPU time that is used to run at the core level (kernel).

%iowait: Displays the percentage of total CPU time that is used to wait for I/O operations.

%steal: The percentage of the hypervisor (hypervisor) that waits for a virtual CPU to serve another virtual process.

%idle: Shows the percentage of CPU idle time that takes up the total CPU time.

1. If the%iowait value is too high, it indicates that the hard disk has an I/O bottleneck

2. If the value of the%idle is high but the system responds slowly, it is possible that the CPU waits for the allocated memory, at which time the memory capacity should be increased

1. If the value of%idle is consistently below 1, the system has a relatively low CPU capacity, indicating that the most necessary resource to be addressed in the system is the CPU.

If you want to view the contents of a binary file test, you would type the following SAR command:

Sar-u-F Test

Inode, file, and other kernel table monitoring

Dentunued: Directory tells the cache how many entries are not being used

FILE-NR: Number of file handles used

INODE-NR: Number of use of index node handles

PTY-NR: The number of Pty used

Memory and swap space monitoring

Kbmemfree: This value is basically the same as the free value in the command, so it does not include buffer and cache space.

Kbmemused: This value is basically the same as the used value in the free command, so it includes buffer and cache space.

%memused: This value is a percentage of kbmemused and total memory (excluding swap).

Kbbuffers and kbcached: These two values are the buffer and cache in the free command.

Kbcommit: Ensure that the current system requires memory (RAM+SWAP) in order to ensure that it does not overflow.

%commit: This value is a percentage of kbcommit and total memory (including swap).

Memory Paging Monitoring

PGPGIN/S: Indicates the number of bytes per second from disk or swap to memory (KB)

PGPGOUT/S: Indicates the number of bytes per second from memory to disk or swap (KB)

FAULT/S: The number of pages per second that the system generates, that is, the sum of the primary and secondary page faults (major + minor)

MAJFLT/S: The number of main pages generated per second.

PGFREE/S: Number of pages per second that are placed in the free queue

PGSCANK/S: Number of pages scanned per second by KSWAPD

PGSCAND/S: Number of pages scanned directly per second

PGSTEAL/S: The number of pages per second that are purged from the cache to meet memory requirements

%vmeff: Percentage of pages purged per second (pgsteal) in total scan page (Pgscank+pgscand)

IO and transfer rate monitoring

TPS: Total I/O transfers per second for physical devices

Rtps: Total amount of data read from physical devices per second

Wtps: The total amount of data written to the physical device per second

BREAD/S: The amount of data read from the physical device per second, in blocks/s

BWRTN/S: The amount of data written to the physical device per second, in blocks/s

Process Queue Length and average load status monitoring

Runq-sz: Length of the run queue (number of processes waiting to run)

Plist-sz: Number of processes (processes) and threads (threads) in the process list

Ldavg-1: Average system load for last 1 minutes (systems load average)

Ldavg-5: System average load over the last 5 minutes

Ldavg-15: System average load over the last 15 minutes

System Exchange activity Information monitoring

PSWPIN/S: Number of swap pages per second system swapped in (Swap page)

PSWPOUT/S: Number of swap pages per second system swapped out (Swap page)

Monitoring of equipment usage

Parameter-P can print out the name of the disk device such as SDA,HDC, if not the parameter-p, the device node may be dev8-0,dev22-0

TPS: Number of times per second from physical disk I/O. Multiple logical requests are merged into one I/O disk request, and the size of one transfer is indeterminate.

RD_SEC/S: Number of Read sectors per second.

WR_SEC/S: Number of Write sectors per second.

Avgrq-sz: The average data size (sector) per device I/O operation.

Avgqu-sz: The average length of the disk request queue.

Await: The average elapsed time of each request, including the request queue wait time, in milliseconds (1 seconds =1000 milliseconds), from the request disk operation to the completion of the system processing.

SVCTM: The average time that the system processes each request, excluding the time consumed in the request queue.

%UTIL:I/O requests account for the percentage of the CPU, the higher the ratio, the more saturated the description.

1. When the value of Avgqu-sz is low, the equipment utilization is higher.

2. When the value of%util is close to 1%, it indicates that the device bandwidth is already full.

1.7 netstat1.7.1 Command Description

The Netstat command is used to display local network links, run ports, routing tables, and other information

1.7.2 parameter Description

netstat [ options]

• -A (All): Displays a list of all valid connection information, including established connections (established), as well as those connections that listen for connection requests (LISTENING), disconnect (close_wait), or are in an online wait state (time_ Wait), etc.
• -T (TCP): Display TCP-related options
• -U (UDP): Show only UDP-related options
• -N: Refuses to display aliases, can show all numbers converted into numbers.
• -L: Lists only the service states that are in Listen (listening)
• -P: Shows the name of the program that established the related link
• -R: Show routing information, routing table, in addition to showing valid routes, also shows the current valid connection
• -E: Display extended information such as UID, etc.
• -S: Statistics according to each protocol
• -C: Executes the netstat command every other fixed time.
• -V: Displays the current active connection, similar to the-N option
• -I: Display information for auto-match interface
• -E: Displays statistical data about Ethernet. The items it lists include the total number of bytes sent, the number of errors, the number of deletes, the number of datagrams, and the number of broadcasts. These statistics have both the number of datagrams sent and the number of datagrams received. This option can be used to count some basic network traffic.

Hint: The status of listen and listening can only be seen with-a or-l

1.7.3 Result Description

• Iface: Represents the interface name of the network device.
• MTU: Represents the maximum Transfer unit in bytes.
• Rx-ok/tx-ok: Indicates how many packets have been received/sent accurately.
• Rx-err/tx-err: Indicates how many errors were generated when the packet was received/sent.
• RX-DRP/TX-DRP: Indicates how many packets were dropped when the packet was received/sent.
• RX-OVR/TX-OVR: Indicates how many packets have been lost due to errors.
• FLG represents the interface tag, where

B already has a broadcast address set up.

L This interface is a loopback device.

M receives all packets (chaotic mode).

N avoid tracking.

O on this interface, disable a R P.

P This is a point-to-dot link.

The R interface is running.

The U interface is in the active state.

Where the value of Rx-err/tx-err, RX-DRP/TX-DRP and RX-OVR/TX-OVR should be 0, if not 0, and very large, then the network quality is definitely a problem, network transmission performance will decline.

• Recv-q: Represents the Receive queue.
• Send-q: Represents the Send queue.
• Local Address: Indicates the native machine name, port
• Foreign Address: Indicates the remote machine name, port
• State: Represents the status, where

LISTEN: In the listening state.
Established: An online situation has been established.
Time_wait: The online is now waiting for the status.

1.8 uptime1.8.1 Command Description

Uptime is mainly used to count the current operating state of the system.

1.8.2 parameter Description

-V Display version

1.8.3 Result Description

• The output information is: The system now time, the system from the last boot to the current run for how long, the system currently has how many users, the system within a minute, 5 minutes, 15 minutes of the average load
• Note: If the load average value is longer than the number of system CPUs, it indicates that the CPU is busy and the load is high, which may affect system performance.

1.9 ps1.9.1 Command Description

The PS command is a Process view command that uses this command to determine which processes are running and running, whether the process is complete, if the process is dead, which processes are consuming too much resources, and so on.

1.9.2 parameter Description

Common parameters:

• -A show all processes (equivalent to-e) (utility)
• -a displays all processes of a terminal, in addition to session leader
• -N ignores selection.
• -D displays all processes, but omits all session leaders (utility)
• -X displays a process that does not control the terminal, and displays the specific path of each command. DX cannot be combined. (utility)
• -p PID process uses CPU time
• -U uid or username Select a valid user ID or user name
• -G gid or groupname displays all processes for the group.
• U username Displays all processes under the user, and displays the detailed path of each command. such as: PS U Zhang; (utility)
• -F is all listed, and is usually used with other options. such as: Ps-fa or Ps-fx and so on.
• -L long format (with fields such as F,wchan,c)
• -J Job Format
• -o user-defined format.
• V display in virtual memory format
• s displayed in signal format
• -M Show All threads
• -H displays the level of the process (shared with other commands, such as: ps-ha) (utility)
• The e command then displays the environment (for example: ps-d e; Ps-a e) (utility)
• H does not display the first line

Common usage:

• PS A: Displays all programs under the current terminal, including other users ' programs.
• PS-A: Shows all programs.
• PS C: When the program is listed, the actual instruction name of each program is displayed, without the indication of the path, parameter, or resident service.
• PS-E: The effect of this parameter is the same as specifying the "A" parameter.
• PS E: Displays the environment variables used by each program when the program is listed.
• PS F: Displays the tree structure with ASCII characters and expresses the relationship between the programs.
• Ps-h: Displays a tree structure that represents inter-program relationships.
• Ps–n: Shows all the programs except the ones under the PS command Terminal.
• PS S: Show program status using program signal format.
• PS S: When listing programs, include interrupted sub-program data.
• ps-t< Terminal number >: Specify the terminal number and list the status of the program that belongs to the terminal.
• PS U: the user-oriented format to display the status of the program.
• PS x: Shows all programs, not the terminal to distinguish.
• Ps-l: Longer and more detailed display of the PID information

The most common approach is to Ps-aux, and then use a pipe symbol to direct to grep to find a specific process and then manipulate the specific process.

1.9.3 Result Description

• User username
• UID Subscriber ID (user ID)
• PID Progress ID (Process ID)
• PPID Process ID (parent process ID)
• SID Conversation ID (session ID)
• CPU utilization of the%CPU process
• Memory utilization of the%MEM process
• The size of the virtual memory used by the VSZ process (virtual size)
• The size of the resident set used by the RSS process, or the size of the actual memory, Kbytes bytes.
• TTY associated with a process terminal (TTY)
• Status of the stat process: process State is represented by a character (stat's status code)
  • R runs Runnable (on Run queue) is running or waiting in the run queue.
  • S sleep sleeping dormant, obstructed, waiting for a condition to form or receive signals.
  • I Free Idle
  • The Z Zombie Zombie (a defunct process) session has been terminated, but the process descriptor exists until the parent process calls the WAIT4 () system call and releases.
  • D non-interruptible uninterruptible sleep (ususally IO) receives a signal that does not wake up and is not operational, and the process must wait until an interrupt occurs.
  • T terminates the Terminate process to receive sigstop, SIGSTP, Sigtin, and Sigtou signals after it stops running.
  • P Wait for Exchange page
  • W no-resident page has no resident pages there is not enough memory paging to allocate.
  • X dead Process
  • < high-priority process
  • N low-priority process with lower priority sequence
  • L Memory Lock page lock has memory paging allocated and shrunk in memory body
  • Leader of the S-process (under it there are sub-processes);
  • L multi-process (using Clone_thread, similar to NPTL pthreads)
  • + process groups in the background
  • Start process startup time and date
  • The total CPU time used by the duration process
  • command-line command being executed
  • NI Priority (Nice)
  • PRI Process Precedence Number (priority)
  • The name of the kernel function in which the Wchan process is sleeping; the name of the function is obtained from the/root/system.map file.
  • FLAGS and process-related digital IDs

1.10 Watch

1.10.1 Command Description

Real-time monitoring commands, as well as commands to detect other command run conditions

1.10.2 parameter Description

-D Highlighting changes

-N Period (seconds)

1.10.3 Result Description

Watch–d–n 1 Netstat-ant

Monitor the network every second, highlighting changes.

1.11 strace1.11.1 Command Description

The Strace command is used to track system calls and received signals while the process is executing. In the Linux world, processes cannot directly access hardware devices, and when a process requires access to a hardware device (such as reading a disk file, receiving network data, and so on), it must be switched from user mode to kernel mode and access to the hardware device through system calls. Strace can trace the system calls generated by a process, including parameters, return values, time spent executing

1.11.2 parameter Description

Strace Using Parameters

• -P: tracks the specified process.
• -F: Trace is called by the fork child process system.
• -F: Attempts to trace the Vfork sub-process system to the inhalation, with-f simultaneous appearance, vfork is not tracked.
• -o FileName: The default strace outputs the result to stdout. The output can be written to the filename file by-O.
• -FF: Often used with the-o option, the system calls produced by different processes (sub-processes) are output to filename. PID text
• -r: Prints the relative time of each system call.
• -T: Add time information before each line in the output. The-TT time is determined to the microsecond level. You can also use-TTT to print relative time.
• -V: Outputs all system calls. By default, some system calls that are called frequently do not output.
• -S: Specifies the length of the output string for each line, which defaults to 32. The file name is always output.
• -C: Counts the time, number of calls, and number of errors performed by each system call.
• -E Expr: Output filters, through expressions, can filter out the output you do not want.
• -D: Output strace debug information about standard errors.
• -H: Outputs a brief help message.
• -I: The entry pointer for the output system call.
• -Q: Suppresses the output of messages about secession.
• -TT: Add time information to microsecond levels before each line in the output.
• -T: Displays the elapsed time for each call.
• -V: Output strace version information.
• -X: Outputs non-standard strings in 16 binary form.
• -XX: All strings are output in 16 binary form.

1.11.3 Result Description

Strace-ff-f-o ls.log ls–l tracking ls–l command execution

When a function fails to execute, the return value is typically-1

1.12 lsof1.12.1 Command Description

The original feature of Lsof is the process of listing open files. All files under Linux.

1.12.2 parameter Description

• -A: Lists the processes that exist for open files
• -c< Process name >: Lists files opened by the specified process
• -G: List GID process details
• -d< File Number >: Lists the processes that occupy the file number
• +d< directory >: List files opened in directory
• +d< directory >: Recursively list files opened in directory
• -n< directory >: List files that use NFS
• -i< condition >: Lists the processes that meet the criteria.
• -p< Process Number: Lists files opened by the specified process number
• -U followed by username: Lists files opened by this user-related process
• -U: List only system socket file types
• -H: Display Help information
• -V: Display version information

1.12.3 Result Description

List the socket file processes under all root users

• COMMAND: Name of the process
• PID: Process Identifier
• USER: Process Owner
• FD: File descriptor in which the application recognizes the file through a file descriptor. such as CWD, TXT, etc.
• Type: File types, such as Dir, Reg, etc.
• DEVICE: Specifies the name of the disk
• Size: Sizes of files
• Node: Index node (the identity of the file on disk)
• Name: Open the exact name of the file

Full coverage of Linux monitoring commands