Linux Performance analysis Tools

Overview

|  1-P all11-xz1--NDEV 1-n tcp,etcp 1top

Uptime

$23:51:26 up 21:31,  1 user,  load average:30.02, 26.43, 19.02

DMESG | Tail

$ DMESG | tail[1880957.563150] Perl invoked Oom-killer:gfp_mask=0x280da, order=0, oom_score_adj=0[...] [process 18694 (perl) score 246 or sacrifice child[process 18694 (perl) TOTAL-VM : 1972392kB, anon-rss:1953348kb, file-rss:0kb[2320864.954447] tcp:possible SYN flooding on port 7001. Dropping request.  Check SNMP counters.

Vmstat 1

$ vmstat 1---------memory-------------Swap-------io-----system--------CPU----- r  B swpd   free< C4/>buff  cache   si   so    bi    bo   in   CS US sy ID WA St  0    0 200889792  73708 591828    0    0     0     5    6   ten  1 3 0 032  0    0 200889920  73708 591860    0    0     0   592 13284 4282 98  1  1  0  032  0    0 200890112  73708 591860    0    0     0     0 9501 2154  1  0 0  0    0 200889568  73712 591856    0    0 0 032    11900 2459  0  0 0 032  0    0 200890208  73712 591860    0    0     0     0 15898 4840 98  1  1  0  0

Mpstat-p all 1

$ mpstat-p All 13.13.0-49-generic (titanclusters-xxxxx)  07/14/2015  _x86_64_ ( CPU) 07:38:49 PM  CPU   %usr  %nice   %sys%iowait   %irq  %soft%steal%guest%gnice  %idle07:38:50 PM  all  98.47   0.00   0.75    0.00   0.00   0.00    0.00    0.00    0.00   0.7807:38:50 PM    0  96.04   0.00   2.97 0.00 0.00 0.00    0.00    0.00    0.00   0.9907:38:50 PM    1  97.00   0.00   1.00    0.00   0.00   0.00    0.00    0.00    0.00   2.0007:38:50 PM    2  98.00   0.00   1.00    0.00   0.00   0.00    0.00    0.00    0.00   1.0007:38:50 PM    3  96.97   0.00   0.00    0.00   0.00   0.00    0.00    0.00    0.00   3.03[...]

Pidstat 1

$ pidstat 1Linux3.13.0-49-generic (titanclusters-xxxxx) 07/14/2015 _x86_64_ (32CPU)07:41:02 PM UID PID%usr%system%guest%CPU CPU Command07:41:03 PM 0 9 0.00 0.94 0.00 0.94 1 rcuos/007:41:03 PM 0 4214 5.66 5.66 0 . xx 11.32 mesos-slave07:41:03 PM 0 4354 0.94 0.94 0.00 1.89 8Java07:41:03 PM 0 6521 1596.23 1.89 0.00 1598.11 27Java07:41:03 PM 0 6564 1571.70 7.55 0.00 1579.25 28Java07:41:03 PM 60004 60154 0.94 4.72 0.00 5.66 9Pidstat07:41:03 PM UID PID%usr%system%guest%CPU CPU Command07:41:04 PM 0 4214 6.00 2.00 0.00 8.00slave07:41:04 PM 0 6521 1590.00 1.00 0.00 1591.00 27Java07:41:04 PM 0 6564 1573.00 10.00 0.00 1583.00 28Java07:41:04 PM 108 6718 1.00 0.00 0.00 1.00 0 snmp-Pass07:41:04 PM 60004 60154 1.00 4.00 0.00 5.00 9Pidstat^c

IOSTAT-XZ 1

$ IOSTAT-XZ 1Linux3.13.0-49-generic (titanclusters-xxxxx) 07/14/2015 _x86_64_ (32CPU) Avg-CPU:%user%nice%system%iowait%steal%Idle73.96 0.00 3.73 0.03 0.06 22.21DEVICE:RRQM/s wrqm/s r/s w/s rkb/s wkb/s avgrq-sz avgqu-sz await r_await w_await SVCTM%Utilxvda0.00 0.23 0.21 0.18 4.52 2.08 34.37 0.00 9.98 13.80 5.42 2.44 0.09Xvdb0.01 0.00 1.02 8.94 127.97 598.53 145.79 0.00 0.43 1.78 0.28 0.25 0.25XVDC0.01 0.00 1.02 8.86 127.79 595.94 146.50 0.00 0.45 1.82 0.30 0.27 0.26DM-0 0.00 0.00 0.69 2.32 10.47 31.69 28.01 0.01 3.23 0.71 3.98 0.13 0.04DM-1 0.00 0.00 0.00 0.94 0.01 3.78 8.00 0.33 345.84 0.04 346.81 0.01 0.00DM-2 0.00 0.00 0.09 0.07 1.35 0.36 22.50 0.00 2.55 0.23 5.62 1.78 0.03[...] ^c

Free-m

$ free-m       Total used       free     shared    buffers     cachedmem:        245998      24545     221453        541-/+ buffers/cache:      23944     222053Swap:            0          0          0

Sar-n DEV 1

$ sar-n DEV 1Linux3.13.0-49-generic (titanclusters-xxxxx) 07/14/2015 _x86_64_ (32CPU)12:16:48 AM IFACE rxpck/s txpck/s rxkb/s txkb/s rxcmp/s txcmp/s rxmcst/s%Ifutil12:16:49 am eth0 18763.00 5032.00 20686.42 478.30 0.00 0.00 0.00 0.0012:16:49 am Lo      14.00 14.00 1.36 1.36 0.00 0.00 0.00 0.0012:16:49 AM DOCKER0 0.00 0.00   0.00 0.00 0.00 0.00 0.00 0.0012:16:49 AM IFACE rxpck/s txpck/s rxkb/s txkb/s rxcmp/s txcmp/s rxmcst/s%Ifutil12:16:50 am eth0 19763.00 5101.00 21999.10 482.56 0.00 0.00 0.00 0.0012:16:50 am Lo      20.00 20.00 3.25 3.25 0.00 0.00 0.00 0.0012:16:50 AM DOCKER0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00^c

Sar-n tcp,etcp 1

$ sar-n tcp,etcp 13.13.0-49-generic (titanclusters-xxxxx)  07/14/2015    _x86_64_    ( CPU) 12:17:19 am  active/s passive/s    iseg/s    oseg/s12:17:20 am      1.00      0.00  10233.00  18846.0012:17:19 AM  atmptf/s  estres/s retrans/s isegerr/s   orsts/s12:17:20 Am      0.00      0.00      0.00      0.00      0.0012:17:20 am  active/s passive/s    iseg/s    oseg/ s12:17:21 am      1.00      0.00   8359.00   6039.0012:17:20 am  atmptf/s  estres/s RETRANS/S isegerr/s   orsts/s12:17:21 AM      0.00      0.00      0.00      0.00      0.00^c 

Top

$Toptop-00:15:40 up 21:56, 1 user, load average:31.09, 29.87, 29.92Tasks:871 Total, 1 running, 868 sleeping, 0 stopped, 2Zombie%CPU (s): 96.8 us, 0.4 sy, 0.0 ni, 2.7 ID, 0.1 wa, 0.0 hi, 0.0 si, 0.0Stkib Mem:25190241+total, 24921688 used, 22698073+free, 60448Bufferskib Swap:0 Total, 0 used, 0 free. 554208cached Mem PID USER PR NI VIRT RES SHR S%CPU%MEM time+COMMAND20248 Root 0 0.227t 0.012t 18748 S 3090 5.2 29,812:58Java4213 Root 0 2722544 64640 44232 S 23.5 0.0 233:35.37 mesos-slave66128 titancl+ 0 24344 2332 1172 R 1.0 0.0 0:00.07Top5235 Root 0 38.227g 547004 49996 S 0.7 0.2 2:02.74Java4299 Root 0 20.015g 2.682g 16836 S 0.3 1.1 33:14.42Java1 root 0 33620 2920 1496 S 0.0 0.0 0:03.82Init2 Root 0 0 0 0 S 0.0 0.0 0:00.02Kthreadd3 Root 0 0 0 0 S 0.0 0.0 0:05.35 ksoftirqd/0 5 root 0-20 0 0 0 S 0.0 0.0 0:00.00 kworker/0: 0H6 Root 0 0 0 0 S 0.0 0.0 0:06.94 kworker/u256:0 8 Root 0 0 0 0 S 0.0 0.0 2:38.05 rcu_sched

Linux Performance analysis Tools