# Identifying system performance bottlenecks # sysdig,

# Identifying system performance bottlenecks # sysdig,

• Installation Method:

Curl-s https://s3.amazonaws.com/download.draios.com/stable/install-sysdig | sudo bash [root @ localhost ~] # Curl-s https://s3.amazonaws.com/download.draios.com/stable/install-sysdig | sudo bash * Detecting operating system * Installing EPEL repository (for DKMS) * Installing Draios public key * Installing Draios repository * Installing kernel headers is omitted ....

Run sysdig directly after installation. Pay attention to pressing ctrl-c at any time after screen flushing.

1187 11:24: data=root:x:0:.bin:x:1:.daemon:x:2:.sys:x:3:.adm:x:4:.tty:x:5:.disk:x:6:.lp:x:7:.mem:1264 11:24:27.109080267 0 systemd-udevd (489) > close fd=14(<f>/etc/group)1265 11:24:27.109080581 0 systemd-udevd (489) < close res=01266 11:24:27.109082572 0 systemd-udevd (489) > munmap addr=7F4CE05B7000 length=40961267 11:24:27.109085958 0 systemd-udevd (489) < munmap res=0 vm_size=45468 vm_rss=1984 vm_swap=12281268 11:24:27.109089928 0 systemd-udevd (489) > read fd=13(<f>/usr/lib/udev/rules.d/69-cd-sensors.rules) size=40961269 11:24:27.109090545 0 systemd-udevd (489) < read res=0 data=1270 11:24:27.109090992 0 systemd-udevd (489) > close fd=13(<f>/usr/lib/udev/rules.d/69-cd-sensors.rules)1271 11:24:27.109091119 0 systemd-udevd (489) < close res=01272 11:24:27.109091753 0 systemd-udevd (489) > munmap addr=7F4CE05B8000 length=40961273 11:24:27.109093095 0 systemd-udevd (489) < munmap res=0 vm_size=45464 vm_rss=1980 vm_swap=12281274 11:24:27.109094748 0 systemd-udevd (489) > stat

It is similar to the output of strace and tcpdump. The output information is as follows:

%evt.num %evt.time %evt.cpu %proc.name (%thread.tid) %evt.dir %evt.type %evt.args

Where:

* Evt. num is the auto-increment event number * evt. time is the timestamp * evt of the event. cpu is the CPU number of the capture event * proc. name is the name of the process that generates the event * thread. tid is the id of the thread that generates the event * evt. dir is the event direction,> indicates that the event is logged <indicates that the event is exited * evt. type is the event name, such as 'open' or 'read' * evt. args event parameter list

• File input and output:

Use the-w option to save the output information as a file:

$ sysdig –w myfile.scap

-N option specifies the number of rows:

$ sysdig –n 100 –w myfile.scap

The-r option indicates reading and outputting data from saved files:

$ sysdig –r myfile.scap

• Filter:

The sysdig filter is a bit like tcpdump. As for the specific meaning of the output format, the filter here can be used to set any item. For example, I want to filter the sshd process:

[root@localhost ~]# sysdig proc.name=sshd 29255 11:44:47.958210052 0 sshd (2131) > clock_gettime29256 11:44:47.958210248 0 sshd (2131) < clock_gettime29257 11:44:47.958211128 0 sshd (2131) > select29258 11:44:47.958212200 0 sshd (2131) < select res=129259 11:44:47.958212428 0 sshd (2131) > rt_sigprocmask29260 11:44:47.958212625 0 sshd (2131) < rt_sigprocmask29261 11:44:47.958213022 0 sshd (2131) > rt_sigprocmask29262 11:44:47.958213187 0 sshd (2131) < rt_sigprocmask29263 11:44:47.958213389 0 sshd (2131) > clock_gettime29264 11:44:47.958213535 0 sshd (2131) < clock_gettime29265 11:44:47.958214129 0 sshd (2131) > write fd=3(<4t>192.168.1.12:4686->10.18.5.47:22) size=266029266 11:44:47.958283315 0 sshd (2131) < write res=2660 data=.r.t..T.=..........{........<).......3.....Y>G..4.M.s..]....Q.1hAN...s.z.....B..29267 11:44:47.958284557 0 sshd (2131) > clock_gettime29268 11:44:47.958284822 0 sshd (2131) < clock_gettime29269 11:44:47.958285308 0 sshd (2131) > select29270 11:44:47.958287711 0 sshd (2131) > switch next=0 pgft_maj=8 pgft_min=1714 vm_size=135212 vm_rss=2320 vm_swap=029272 11:44:47.958519093 0 sshd (2131) < select res=129273 11:44:47.958520932 0 sshd (2131) > rt_sigprocmask29274 11:44:47.958521450 0 sshd (2131) < rt_sigprocmask29275 11:44:47.958521687 0 sshd (2131) > rt_sigprocmask29276 11:44:47.958521860 0 sshd (2131) < rt_sigprocmask 

This is a very common filtering method. Generally, we are interested in specific processes. After specifying the process name, we can capture all system call events.

Similar to tcpdump, sysdig also supports the following comparison symbols and Boolean symbols:

Standard comparison operators (= ,! =, <, <=, >,>=, Contains)
Boolean operators (and, or and not)

$ sysdig proc.name=cat or proc.name=vi

Capture System calls of cat or vi processes. or is used here.

$ sysdig proc.name!=cat and evt.type=open

Show all open calls except cat Processes

$ Sysdig-l -------------------- Field Class: fdfd. num the unique number identifying the file descriptor. fd. type of FD. can be 'file', 'Directory ', 'ipv4', 'ipv6 ', 'unix', 'pipe', 'event', 'signalfd ', 'eventpoll ', 'inotify 'or 'signal fd '. fd. typechar type of FD as a single character. can be 'F' for file, 4 for IP v4 socket, 6 for IPv6 socket, 'U' for unix socket, p for pipe, 'E' for eventfd ,' S 'for signalfd, 'l' for eventpoll,' I 'for I then y, 'O' for uknown. fd. name FD full name. if the fd is a file, this field contains the full path. if the FD is a socket, this field contain the connection tuple. fd. directory If the fd is a file, the directory that contains it. fd. filename If the fd is a file, the filename without the path. fd. ip matches the ip address (client or server) of the fd. fd. c Ip client IP address. fd. sip server IP address. fd. lip local IP address. omitted ....

-L option will list all available fields. We can use the fields shown here to match the required information.

Make good use of these practical fields to monitor various system calls based on your actual situation,
For example, you want to capture the network connections of the Local Machine other than nginx:

$ sysdig evt.type=accept and proc.name!=nginx

• Evt. arg and evt. rawarg

These two fields need to be explained in particular. The events captured by sysdig generally belong to some type (such as open \ read) and a series of parameters (such as fd \ name ).

The powerful feature of sysdig is that you can use a filter to filter parameters.

$ sysdig evt.type=execve and evt.arg.ptid=bash

This filter filters execve system calls of the parent process named bash. The difference between evt. arg and event. rawarg is that the latter does not parse PID, FD, error codes, etc.

$ sysdig evt.arg.res=ENOENT

This can be used to capture a specific I/O error. This value is usually negative.

$ sysdig " evt.rawarg.res<0 or evt.rawarg.fd<0"

It will give you all system calls that generate errors.

• Obtain all system calls

$ Sysdig-L output is omitted...

• View available chisel lists

$ sysdig -cl | less

By default, the following categories are available, and each category contains multiple built-in chisel.

* CPU Usage: CPU Usage * Errors: Error * I/O * Logs: log * Misc: Hybrid * Net: Network * Performance: Performance * Security: Security * System State: System status

• Sysdig instance:

We may often encounter network I/O monitoring. For sysdig, it is easy to perform communication sniffing, and its style is more user-friendly.

For example, you can check the data (in ASCII format) provided by a specific IP address or a specific process (such as nginx ):

# sysdig -s 4096 -A -c echo_fds fd.cip=192.168.1.100 -r /tmp/sysdig-scap.gz proc.name=nginx

If you want to monitor native data transmission (in binary format), replace "-A" with "-X ":

# sysdig -s 4096 -X -c echo_fds fd.cip=192.168.1.100 -r /tmp/sysdig-scap.gz proc.name=nginx

Copyright Disclaimer: This article is an original article by the blogger and cannot be reproduced without the permission of the blogger.