Linux system Debugging Tools Sysdig use of detailed

Sysdig is the Sysdig cloud developed a powerful tool based on the Lua language, as described on the site's home page, "Sysdig is open source, System-level exploration:capture system State and activity from a running Linux instance, then save, filter and analyze. The IT as Strace + tcpdump + lsof + awesome sauce. ", Sysdig is equivalent to Strace + tcpdump + lsof + htop + iftop and other tools of the collection, in addition to its containers such as Docker, CoreOS, LXC monitoring, is not the feeling of the big leather blowing? What is the function, look below.

Note: In addition, the tool also supports Windows platform and Mac platform, because the work is mainly related to Linux platform, interested in can test themselves.

First, installation

This article mainly introduces the usage installation and test under CentOS, and also mentions the installation under the Ubuntu class platform. Its support is installed on CENTOS6 and Centos7, and the installation method is simple:

1, CentOS under the installation

1.1 One-click Installation

Curl-s Https://s3.amazonaws.com/download.draios.com/stable/install-sysdig | sudo bash
Above is a shell script that identifies the commonly used Linux distributions, configures the source according to the corresponding version, and finally installs the Sysdig package. The Epel source is configured first on the Redhat/centos to install the dkms package, and the Draios source can be configured to install the Sysdig package. Finally, the Kernel-devel package is installed before the Sysdig package is loaded.

1.2, step-by-Step installation

Due to the use of the source are foreign sources, there will be slower installation, so the use of a key installation failure, you can use the decomposition step to install. In the intranet environment can also be the dependency pack down and then install.

#导入draios源
RPM--import Https://s3.amazonaws.com/download.draios.com/DRAIOS-GPG-KEY.public
Curl-s-o/etc/yum.repos.d/draios.repo Http://download.draios.com/stable/rpm/draios.repo
#导入epel Source
Rpm-i http://mirror.us.leaseweb.net/epel/6/x86_64/epel-release-6-8.noarch.rpm
#装包
Yum-y Install kernel-devel* dkms sysdig
Sysdig relies on two packages, one is Kernel-devel, the other is the dkms package, dkms full name is Dynamic Kernel module Support (supported by dynamical kernel modules), that is, the new driver module can be rebuilt automatically after the kernel version changes. What you want to know is Google.

2, the installation under the Ubuntu

Curl-s Https://s3.amazonaws.com/download.draios.com/DRAIOS-GPG-KEY.public | Apt-key Add-
Curl-s-o/etc/apt/sources.list.d/draios.list Http://download.draios.com/stable/deb/draios.list
Apt-get Update
Apt-get-y Install linux-headers-$ (uname-r)
Apt-get-y Install Sysdig
Windows, Mac OS, and other Linux distributions can be installed in the official installation documentation.

Ii. Common usage

By default, when the above method is installed, the execution of Sysdig will be wrong. The prompts are as follows:

# Sysdig
Unable to load the driver
Error opening device/dev/sysdig0. Make sure your have root credentials and that's Sysdig-probe module is loaded.
So you need to use the/usr/bin/sysdig-probe-loader command to load the kernel module before execution, which is also a shell script that downloads an KO module file from the AWS S3 when it executes. However, I failed to download the Linux test host many times. After viewing the script file, it is found that its call to download the address is: https://s3.amazonaws.com/download.draios.com/stable/sysdig-probe-binaries/ Sysdig-probe-0.6.0-x86_64-2.6.32-504.el6.x86_64-e065a96a1a7343d57e26548de23096e3.ko

Note: The Ko file URL is not to be remembered, the script will be executed with the corresponding prompt "trying to download precompiled module from", and different kernel versions of the corresponding KO files are different. My kernel version here is 2.6.32-504.el6. When the download is complete, it is stored in the ~/.sysdig directory.

After the completion of the Sysdig-probe-loader command can execute the sysdig command, and will not automatically load after the boot, so in the absence of use, the package is not affected by the host.

1, Network

To view the processes that occupy the most network bandwidth:

Sysdig-c topprocs_net
Displays network transport data for host 192.168.0.1:

As binary:
Sysdig-s2000-x-C Echo_fds fd.cip=192.168.0.1
As ASCII:
Sysdig-s2000-a-C Echo_fds fd.cip=192.168.0.1
This shows the network transmission function, the actual effect and tcpdump grab bag is the same. And it also supports sysdig-w Dump.scap grab pack (can be configured with-X or-a use), the bag also supports Sysdir-r Dump.scap read.

To view the most connected server ports:

In terms of established connections:
Sysdig-c fdcount_by fd.sport "Evt.type=accept"
In terms of total bytes:
Sysdig-c fdbytes_by Fd.sport
To view the IP with the most client connections:

In terms of established connections
Sysdig-c fdcount_by Fd.cip "Evt.type=accept"
In terms of total bytes
Sysdig-c fdbytes_by Fd.cip
Lists all access connections that are not accessing the Apache service:

Sysdig-p "%proc.name%fd.name" "Evt.type=accept and Proc.name!=httpd"
2. Hard disk I/O

To view the processes that use the most disk bandwidth:

Sysdig-c Topprocs_file
Lists processes that use a large number of file descriptors

Sysdig-c fdcount_by proc.name "Fd.type=file"
The top files in terms of read+write bytes

Sysdig-c topfiles_bytes
Print the top files this Apache has been reading from or writing to

Sysdig-c topfiles_bytes proc.name=httpd
Basic Opensnoop:snoop file opens as they occur

Sysdig-p "%12user.name%6proc.pid%12proc.name%3fd.num%fd.typechar"%fd.name
Directories in terms of r+w disk activity

Sysdig-c fdbytes_by fd.directory "Fd.type=file"
The top files in terms of r+w disk activity in The/tmp directory

Sysdig-c fdbytes_by fd.filename "fd.directory=/tmp/"
Observe the I/O activity in all the files named ' passwd '

Sysdig-a-C Echo_fds "FD.FILENAME=PASSWD"
Display I/O activity by FD type

Sysdig-c fdbytes_by Fd.type

3. Process and CPU utilization

Processes in terms of CPU usage

Sysdig-c TOPPROCS_CPU
Processes for CPU 0

Sysdig-c topprocs_cpu evt.cpu=0
Observe the standard output of a process

Sysdig-s4096-a-C stdout Proc.name=cat

4. Application

View all HTTP requests for the machine

sudo sysdig-s 2000-a-c Echo_fds fd.port=80 and Evt.buffer contains get
View all SQL Select queries for the machine

sudo sysdig-s 2000-a-c Echo_fds Evt.buffer contains SELECT
Queries made via Apache to a external MySQL server happening in real time

Sysdig-s 2000-a-C Echo_fds fd.sip=192.168.30.5 and Proc.name=apache2 and Evt.buffer contains SELECT
5, Performance and error

The files where most time has been spent

Sysdig-c Topfiles_time
The files where Apache spent most time

Sysdig-c Topfiles_time proc.name=httpd
Processes in terms of I/O errors

Sysdig-c topprocs_errors
The top files in terms of I/O errors

Sysdig-c topfiles_errors
All the failed disk I/O calls

Sysdig Fd.type=file and Evt.failed=true
The failed file opens by httpd

Sysdig "proc.name=httpd and Evt.type=open and Evt.failed=true"
The system calls where most time has been spent

Sysdig-c Topscalls_time
The top system calls returning errors

Sysdig-c topscalls "Evt.failed=true"
Snoop failed file opens as they occur

Sysdig-p "%12user.name%6proc.pid%12proc.name%3fd.num%fd.typechar%fd.name" Evt.type=open and Evt.failed=true
Print the file I/O calls that have a latency greater than 1ms:

Sysdig-c Fileslower 1
6. Safety

Show the directories the user "root" visits

Sysdig-p "%evt.arg.path" "Evt.type=chdir and User.name=root"
Observe SSH activity

Sysdig-a-C Echo_fds fd.name=/dev/pretmx and Proc.name=sshd
Show every file open that happens in/etc

Sysdig Evt.type=open and Fd.name contains/etc
Show the ID of the ' all ' login shells that have launched the ' tar ' command

Sysdig-r file.scap-c List_login_shells Tar
Show all the commands executed by the login shell with the given ID

Sysdig-r trace.scap.gz-c spy_users proc.loginshellid=5459
7. Container

To view a list of containers running on a machine and its resource usage

sudo csysdig-vcontainers
To view the list of processes for a container context

sudo csysdig-pc
View CPU usage running in WORDPRESS1 container

sudo sysdig-pc-c topprocs_cpu container.name=wordpress1
View usage of network bandwidth running in WORDPRESS1 containers

sudo sysdig-pc-c topprocs_net container.name=wordpress1
View the processes that use the most network bandwidth in the WORDPRESS1 container

sudo sysdig-pc-c topprocs_net container.name=wordpress1
View the files that occupy the most I/O bytes in the Wordpress1 container

sudo sysdig-pc-c topfiles_bytes container.name=wordpress1
View the ranking of network Connections in WORDPRESS1 containers

sudo sysdig-pc-c Topconns container.name=wordpress1
Shows the execution of all commands in the Wordpress1 container

sudo sysdig-pc-c spy_users container.name=wordpress1
The function is not very powerful, but it is a bit cumbersome to remember, you can use Sysdig-l to view the list of supported events, using SYSDIG-L to view the filtered list of events supported by the event. You can also refer to the Official Guide documentation.

Third, Csysdig interactive processing

Feel the above method is still more troublesome how to do? Can you not choose interactively? Of course, the Sysdig package also provides a tool csysdig that, after the tool is executed, has the same effect as the top command