#定位系统性能瓶颈 # Sysdig

• Installation method:

Curl-s Https://s3.amazonaws.com/download.draios.com/stable/install-sysdig | sudo bash[[email protected] ~]# curl-s Https://s3.amazonaws.com/download.draios.com/stable/install-sysdig | sudo bash* detecting operating system* installing EPEL repository (for DKMS) * Installing Draios public key* Installing Dra iOS repository* installing kernel headers omitted ....

After installation apart directly run Sysdig, notice the brush screen at any time press Ctrl-c

1187 11:24:data=root:x:0:.bin:x:1:.daemon:x:2:.sys:x:3:.adm:x:4:.tty:x:5:.disk:x:6:.lp:x:7:.mem:1264 11:24:27.109080267 0 systemd-udevd (489) > Close fd=14 (<f>/etc/group) 1265 11:24:27.109080581 0 SYSTEMD-UDEVD ( 489) < close res=01266 11:24:27.109082572 0 systemd-udevd (489) > Munmap addr=7f4ce05b7000 length=40961267 11:24:27. 109085958 0 SYSTEMD-UDEVD (489) < Munmap res=0 vm_size=45468 vm_rss=1984 vm_swap=12281268 11:24:27.109089928 0 systemd- UDEVD (489) > Read fd=13 (<f>/usr/lib/udev/rules.d/69-cd-sensors.rules) size=40961269 11:24:27.109090545 0 SYSTEMD-UDEVD (489) < read res=0 data=1270 11:24:27.109090992 0 systemd-udevd (489) > Close fd=13 (<f>/usr/lib/ Udev/rules.d/69-cd-sensors.rules) 1271 11:24:27.109091119 0 SYSTEMD-UDEVD (489) < close res=01272 11:24:27.109091753 0 systemd-udevd (489) > Munmap addr=7f4ce05b8000 length=40961273 11:24:27.109093095 0 systemd-udevd (489) < Munmap R Es=0 vm_size=45464 vm_rss=1980 vm_swap=12281274 11:24:27.109094748 0 systemd-udevd (489) > Stat 

Somewhat similar to the output of Strace, tcpdump, whose output information means:

%evt.num%evt.time%evt.cpu%proc.name (%thread.tid)%evt.dir%evt.type%evt.args

which

* Evt.num is the self-increment event number * Evt.time is the timestamp that the event occurred * EVT.CPU is the CPU number where the event was captured * Proc.name is the process name that generated the event * Thread.tid is the thread that generated the event id* Evt.dir is the event direction, > means Enter Event < indicates Exit event * Evt.type is the event name, such as ' open ' or ' read ' Evt.args event parameter list

• File input/output:

Use the-W option to save the output information as a file:

$ sysdig–w Myfile.scap

The-n option specifies the number of rows:

$ sysdig–n 100–w Myfile.scap

The-r option indicates that the saved file is read out and output:

$ sysdig–r Myfile.scap

• Filter:

Sysdig filter is a bit like tcpdump, in front of the specific meaning of the output format, the filter here can be any one of the settings, such as I filter the sshd process:

[[email protected] ~]# sysdig proc.name=sshd 29255 11:44:47.958210052 0 sshd (2131) > clock_gettime29256 11:44:47 .958210248 0 sshd (2131) < clock_gettime29257 11:44:47.958211128 0 sshd (2131) > select29258 11:44:47.958212200 0 SS HD (2131) < select res=129259 11:44:47.958212428 0 sshd (2131) > rt_sigprocmask29260 11:44:47.958212625 0 sshd (2131  ) < rt_sigprocmask29261 11:44:47.958213022 0 sshd (2131) > rt_sigprocmask29262 11:44:47.958213187 0 sshd (2131) < rt_sigprocmask29263 11:44:47.958213389 0 sshd (2131) > clock_gettime29264 11:44:47.958213535 0 sshd (2131) < Clock_ gettime29265 11:44:47.958214129 0 sshd (2131) > Write fd=3 (<4t>192.168.1.12:4686->10.18.5.47:22) size= 266029266 11:44:47.958283315 0 sshd (2131) < write res=2660 data=.r.t: T.= ..... {........<) .... 3.....y>g. 4.m.s..] .... Q.1han...s.z ..... B.. 29267 11:44:47.958284557 0 sshd (2131) > clock_gettime29268 11:44:47.958284822 0 sshd (2131) < clock_gettime29269 11 :44:47.958285308 0 sshd (2131) > select29270 11:44:47.958287711 0 sshd (2131) > switch next=0 pgft_maj=8 pgft_min=171 4 vm_size=135212 vm_rss=2320 vm_swap=029272 11:44:47.958519093 0 sshd (2131) < select res=129273 11:44:47.958520932 0 S SHD (2131) > rt_sigprocmask29274 11:44:47.958521450 0 sshd (2131) < rt_sigprocmask29275 11:44:47.958521687 0 sshd (2  131) > rt_sigprocmask29276 11:44:47.958521860 0 sshd (2131) < Rt_sigprocmask

This is a very common way of filtering, generally we are interested in a particular process, specifying the process name can be captured to all system call events.

Similar Tcpdump,sysdig also supports the following comparison symbols and Boolean symbols:

Standard comparison operators (=,! =, <, <=, A;, >=, contains)
Boolean operators (and, OR and not)

$ sysdig Proc.name=cat or Proc.name=vi

Capture system calls to cat or VI processes, which are used here or

$ sysdig Proc.name!=cat and Evt.type=open

Show open calls in addition to cat processes

$ sysdig-l----------------------Field Class:fdfd.num The unique number identifying the file Descriptor.fd.type Type of FD.  Can be ' file ', ' directory ', ' IPv4 ', ' IPv6 ', ' UNIX ', ' Pipe ', ' event ', ' SIGNALFD ', ' eventpoll ', ' inotify ' or ' Signal fd '. Fd.typechar type of FD as a single character.                Can is ' f ' for file, 4 for IP v4 sockets, 6 for IPV6 sockets, ' U ' for UNIX sockets, p for pipe, ' E ' for eventfd, ' s ' for SIGNALFD, ' l ' for eventpoll, ' I ' for I notify, ' O ' for Uknown.fd.name FD Full name. If The FD is a file, this field contains the full path. If the FD is a socket, this field contain the connection Tuple.fd.directory If the FD is a file, the Di  Rectory that contains It.fd.filename If the FD is a file, the filename without the Path.fd.ip matches the IP         Address (client or server) of the FD.FD.CIP client IP Address.fd.sip Server IP address.fd.lip local IP address. Omit .... 

The-l option lists all the fields that can be used, and we can use the fields shown here to match the information we need.

Make good use of these practical fields can be based on their own situation to do a variety of system call monitoring,
For example, if you want to capture the local network links other than Nginx can do this:

$ sysdig evt.type=accept and Proc.name!=nginx

• Evt.arg and Evt.rawarg

These two fields need to be specifically explained that the events captured by Sysdig are generally of a certain type (such as open \ Read) and a series of parameters such as FD \ name.

The Sysdig is powerful because filters can be used to filter parameters

$ sysdig Evt.type=execve and Evt.arg.ptid=bash

This filter indicates that the EXECVE system call to the parent process name bash is filtered, and the difference between Evt.arg and Event.rawarg is that the latter does not parse the PID, fd,error codes, and so on.

$ sysdig Evt.arg.res=enoent

This can be used to capture a specific I/O error, which is usually a negative number

$ sysdig "evt.rawarg.res<0 or evt.rawarg.fd<0"

will give you all the system calls that generate the error.

• Get all the system calls

$ sysdig–l Output omitted ...

• View the list of available chisels

$ SYSDIG-CL | Less

The following classes are available by default, with multiple built-in chisels distributed across the categories.

* CPU USAGE:CPU Usage * Errors: Error * i/o* Logs: Log * Misc: Promiscuous * Net: Network * Performance: Performance * Security: Secure * System State: System Status

• Sysdig instances:

We may often encounter monitoring network I/O, and for sysdig, communication sniffing can be easy, and its style is more user-friendly.

For example, you can check the data provided by a specific IP address, such as Nginx, in ASCII encoded format:

# sysdig-s 4096-a-C Echo_fds fd.cip=192.168.1.100-r/tmp/sysdig-scap.gz Proc.name=nginx

If you want to monitor the native data transfer (binary format), replace "-A" with "-X":

# sysdig-s 4096-x-C Echo_fds fd.cip=192.168.1.100-r/tmp/sysdig-scap.gz Proc.name=nginx

Copyright NOTICE: This article for Bo Master original article, without Bo Master permission not reproduced.

#定位系统性能瓶颈 # Sysdig