Page super slow, Zabbix but not the police

Failure background:
Web pages open very slowly

Troubleshooting process:

1. Start with Vmstat see procs under the R value stable at 5, 6, because this server is 12 core 24 threads, and the CPU of WA is very large, it is easy to explain the system,

Certainly won't call the police, then why is the page open so slowly?

[[Email PROTECTED]_SZ nginx]# Vmstat 2
procs-----------Memory-------------Swap-------io------System-------CPU-----
R b swpd free buff cache si so bi bo in CS us sy ID WA St
4 0 141816 4319000 513996 17482000 0 0 0 18 0 0 12 0 88 0 0
2 0 141816 4317912 513996 17482064 0 0 0 460 2118 1581 3 0 97 0 0
0 0 141816 4300932 513996 17482100 0 0 0 1848 1966 1482 3 0 97 0 0
4 0 141816 4308788 513996 17482056 0 0 0 358 1653 1247 2 0 97 0 0
1 0 141816 4314516 513996 17482080 0 0 0 1588 2311 1613 3 0 96 0 0
0 0 141816 4312296 514004 17482020 0 0 0 82952 2122 1353 2 0 98 0 0
3 0 141816 4301508 514004 17482080 0 0 0 408 1880 1424 3 0 97 0 0

[[EMAIL PROTECTED]_SZ nginx]# top
top-16:57:00 up, 10:10, 8 users, load average:4.44, 4.62, 4.61
tasks:613 Total, 2 running, 611 sleeping, 0 stopped, 0 zombie
Cpu (s): 21.4%us, 3.6%sy, 0.0%ni, 75.0%id, 0.0%wa, 0.0%hi , 0.0%si, 0.0%st
mem:24729804k Total, 20219216k used, 4510588k free, 513968k buffers
swap:31457272k Total, 141816k Used, 31315456k free, 17306056k cached

PID USER PR NI VIRT RES SHR S%cpu%MEM time+ COMMAND
6571 Root 0 2920m 384m 348m S 407.1 1.6 83:28.58 Mongod
8514 nginx 0 219m 18m 5328 R 100.0 0.1 0:23.94 php-fpm
9479 Root 0 15432 1672 948 R 81.4 0.0 0:00.08 Top
1 Root 0 19364 1364 1152 S 0.0 0.0 0:02.07 Init
2 Root 0 0 0 0 S 0.0 0.0 0:00.18 Kthreadd
3 root RT 0 0 0 0 S 0.0 0.0 1:00.97 migration/0
4 Root 0 0 0 0 S 0.0 0.0 0:29.55 ksoftirqd/0
5 root RT 0 0 0 0 S 0.0 0.0 0:00.00 migration/0
6 root RT 0 0 0 0 S 0.0 0.0 0:03.90 watchdog/0
7 Root RT 0 0 0 0 S 0.0 0.0 0:20.17 MIGRATION/1
8 root RT 0 0 0 0 S 0.0 0.0 0:00.00 MIGRATION/1
9 Root 0 0 0 0 S 0.0 0.0 0:12.43 KSOFTIRQD/1
Root RT 0 0 0 0 S 0.0 0.0 0:02.43 WATCHDOG/1
Root RT 0 0 0 0 S 0.0 0.0 0:06.51 MIGRATION/2

2. See top ranked first in the Mongod is 400+%, or consumes a certain CPU, because Linux is a multi-threaded overlay processing, so 24 threads

The lower 400+d CPU is actually not particularly high, but this mongod to see, and sure enough fault appeared this mongod, Chen Xujing's over the two days

Enabled a debug thing, has been writing the log, the cumulative run a few days after the mongod slowed down, resulting in the entire business is slow, a small tip, top return

Press 1 to see the load on all CPU processes.

[Email PROTECTED]_SZ nginx]# Top
Top-17:16:40 up, 10:29, 8 users, load average:1.39, 0.87, 1.97
tasks:613 Total, 2 running, 611 sleeping, 0 stopped, 0 zombie
Cpu0:36.2%us, 0.7%sy, 0.0%ni, 63.1%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
Cpu1:46.8%us, 0.3%sy, 0.0%ni, 52.8%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
Cpu2:23.6%us, 0.3%sy, 0.0%ni, 76.1%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
Cpu3:2.3%us, 0.3%sy, 0.0%ni, 97.3%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
Cpu4:0.3%us, 0.3%sy, 0.0%ni, 99.3%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
Cpu5:65.4%us, 0.0%sy, 0.0%ni, 34.6%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
Cpu6:44.7%us, 0.3%sy, 0.0%ni, 55.0%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
Cpu7:53.6%us, 0.7%sy, 0.0%ni, 45.7%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
Cpu8:58.3%us, 0.3%sy, 0.0%ni, 41.4%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
Cpu9:4.0%us, 0.3%sy, 0.0%ni, 95.7%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
Cpu10:10.6%us, 0.0%sy, 0.0%ni, 89.4%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
Cpu11:35.4%us, 0.0%sy, 0.0%ni, 64.6%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
Cpu12:5.0%us, 0.3%sy, 0.0%ni, 94.7%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
Cpu13:0.3%us, 0.0%sy, 0.0%ni, 99.7%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
Cpu14:3.3%us, 0.0%sy, 0.0%ni, 96.4%id, 0.3%wa, 0.0%hi, 0.0%si, 0.0%st
Cpu15:0.3%us, 0.3%sy, 0.0%ni, 99.3%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
Cpu16:0.0%us, 0.0%sy, 0.0%ni,100.0%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
Cpu17:0.0%us, 0.0%sy, 0.0%ni,100.0%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
Cpu18:2.3%us, 0.3%sy, 0.0%ni, 97.4%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
Cpu19:1.7%us, 0.0%sy, 0.0%ni, 98.3%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
Cpu20:3.0%us, 0.0%sy, 0.0%ni, 97.0%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
Cpu21:0.0%us, 0.3%sy, 0.0%ni, 99.7%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
Cpu22:0.0%us, 0.0%sy, 0.0%ni,100.0%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
Cpu23:0.0%us, 0.0%sy, 0.0%ni,100.0%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
mem:24729804k Total, 20473464k used, 4256340k free, 514012k buffers
swap:31457272k Total, 141816k used, 31315456k free, 17502960k cached

Page super slow, Zabbix but not the police