Just when trying to reproduce a bug, it seems that IO has been very high after killed MySQL for some time, as follows:
12:40:01 pm CPU%user%nice%system%iowait%steal%idle12:50:01 PM all 12.86 0.00 14.40 1.58 0.00 71.1601:00:01 PM All 13.38 0.00 15.34 1.50 0.00 69.7901:10:0 1 pm All 34.34 0.00 21.24 2.13 0.00 42.2901:20:01 PM All 36.03 0.00 22.13 3.45 0.00 38.4001:30:01 pm All 36.80 0.00 21.43 2.53 0.00 39.2401:40:01 PM All 36.86 0.00 20.56 2.26 0.00 40.3201:50:01 PM All 38.22 0.00 19.26 2. 0.00 40.4102:00:01 pm All 36.12 0.00 20.52 1.80 0.00 41.5602:10:02 PM All 43.75 0.00 20.05 1.97 0.00 34.2302:20:01 PM All 39.93 0.00 19.16 2.10 0.00 38.8102:30:01 pm All 43.93 0.00 19.38 5.43 0.00 31.2602:40:01 PM All 40.2 7 0.00 21.20 2.21 0.00 36.3202:50:01 pm All 39.17 0.00 21.56 2.10 0.00 37.1603:00:01 PM All 48.89 0.00 19.51 4.21 0.00 27.3803:10:01 PM All 25.04 0.00 16.64 13.96 0.00 44.3703:20:01 pm All 13.49 0.00 18.75 15.09 0.00 52.6703:30:01 PM All 12.69 0.00 17.68 15.35 0.00 54.2703:40:01 PM All 17.22 0.00 13.42 15.50 0 .53.8603:50:01 pm All 19.16 0.00 10.48 14.86 0.00 55.4904:00:01 PM All 11.95 0.00 14.80 15.52) 0.00 57.73
Iotop a bit, the JBD2 process consumes a lot of IO processing, searching for the next, about Jbd2,jbd2 is part of the Ext4 file system. EXT4 file System This bug,bug principle is roughly that the write and request of a file will cause the value of one of the int to increase and eventually increase beyond its own range-to become negative. Will trigger the bug and it will not be easy to reach this value, which can take several months to appear.
Workaround:
1, yum upgrade kernel, restart to see if it is valid. (Prior to this, you should prepare for the use of the standby machine)
2. Reinstall the system partition and mount the data partition after completion.
3, verify the availability of temporary patches. And in the current network repair.
Reference: http://www.361way.com/ext4-jbd2-io-bug/2963.html
CENTOS6 the JBD2 process consumes a lot of IO processing