Optimize synchronization speed for MySQL slave

Test environment:
Red Hat Enterprise Linux Server release 6.3 (Santiago)
Server Version:5.6.22-log MySQL Community Server (GPL)

I built 1 Master 3 from the environment, ready to test the MHA architecture, the process found that when testing concurrent insertions, from the library 1 can keep up, from the library 2,3 keep up with

How to determine whether the IO thread is slow or SQL thread is slow, there is a way to observe show slave status\g,

Determine 3 parameters (the value following the parameter is the normal time of the default idle):
Slave_io_state:waiting for Master to send event
Slave_sql_running_state:slave have read all relay log; Waiting for the slave I/O thread to update it
seconds_behind_master:0

1.sql Thread Slow performance:
Seconds_behind_master is getting bigger.
Slave_sql_running_state:reading event from the relay log

2.io Thread Slow performance:
Seconds_behind_master is 0
Slave_sql_running_state: Show normal values
Slave_io_state: Show Busy status

And the value I observed was
Slave_io_state:waiting for Master to send event
seconds_behind_master:313
Slave_sql_running_state:reading event from the relay log
So the inference is that SQL thread is slow

Why only slave2,3 slow, and slave1 can keep up with it, began to suspect is the difference in parameter configuration, compared to the/etc/my.cnf found after the configuration no difference
Therefore ruled out this reason, and later with Dstat observation, found busy time, slave io write speed does not go
SLAVE1:
$ dstat
----total-cpu-usage-----dsk/total--net/total----paging-----system--
USR sys IDL Wai hiq siq| Read writ| Recv send| In-out | int CSW
0 0 0 0 0|9308b 11k| 0 0 | 3 b 3b| 63 63
3 4 54 40 0 0| 88k 10m| 45k 9438b| 0 0 |1857 2579
3 3 59 35 0 1| 80k 7552k| 40k 8486b| 0 0 |1675 2307
3 3 56 38 0 0| 72k 7824k| 42k 8816b| 0 0 |1727 2348
3 4 52 41 0 1| 96k 9688k| 49k 10k| 0 0 |2029 2874
3 4 54 39 0 0| 96k 8880k| 45k 9410b| 0 0 |1905 2674
3 3 53 40 0 1| 96k 9776k| 58k 10k| 0 0 |1935 2671
3 3 58 36 0 0| 64k 7848k| 40k 8420b| 0 0 |1724 2357
3 5 52 40 0 1| 96k 8936k| 49k 10k| 0 0 |1948 2680
3 4 51 42 0 1| 96k 9400k| 49k 10k| 0 0 |1988 2760
3 4 52 41 0 0| 88k 9752k| 49k 10k| 0 0 |2058 2868
4 4 51 41 0 1| 96k 9680k| 49k 9938b| 0 0 |1990 2750
3 3 59 35 0 0| 80k 7632k| 39k 8288b| 0 0 |1668 2275
3 4 52 42 0 1| 80k 8504k| 46k 9146b| 0 0 |1860 2523
3 4 51 42 0 0| 80k 8496k| 43k 8684b| 0 0 |1882 2516
2 3 65 30 0 0| 64k 5976k| 30k 6440b| 0 0 |1326 1802
3 4 53 40 0 1| 72k 8360k| 59k 10k| 0 0 |1859 2538
3 4 51 42 0 1| 96k 8840k| 53k 10k| 0 0 |1958 2648
2 4 51 43 0 0| 72k 7352k| 40k 7760b| 0 0 |1633 2219
3 4 51 42 0 1| 88k 7920k| 31k 6770b| 0 0 |1767 2373
3 3 54 40 0 0| 80k 8528k| 40k 8750b| 0 0 |1859 2549

Slave2:
----total-cpu-usage-----dsk/total--net/total----paging-----system--
USR sys IDL Wai hiq siq| Read writ| Recv send| In-out | int CSW
2 1 0 1|8192b 1168k| 55k 10k| 0 0 | 533 771
1 1 Wuyi 0 0|8192b 1048k| 33k 7046b| 0 0 | 427 622
1 1 Wuyi 0 0|8192b 1080k| 58k 9806b| 0 0 | 500 709
1 1 50 48 0 0| 0 1864k| 51k 8486b| 0 0 | 502 669
1 2 Wuyi 0 0|8192b 1120k| 42k 8156b| 0 0 | 496 674
1 1 Wuyi 0 0|8192b 1160k| 32k 6350b| 0 0 | 467 655
1 2 51 47 0 0| 0 1288k| 50k 10k| 0 0 | 563 797
1 1 Wuyi 0 0|8192b 1200k| 43k 8486b| 0 0 | 493 728
2 1 0 0|8192b 1024k| 45k 8816b| 0 0 | 481 659
1 1 0 0|8192b 1248k| 49k 9450b| 0 0 | 517 772
1 1 50 48 0 0| 0 1264k| 47k 9146b| 0 0 | 516 756
1 2 0 1|8192b 1144k| 50k 10k| 0 0 | 520 765
1 1 Wuyi 0 0|8192b 1200k| 51k 8156b| 0 0 | 484 716
1 2 0 0|8192b 968k| 50k 9278b| 0 0 | 470 684
1 1 0 0|8192b 1128k| 39k 7892b| 0 0 | 476 679
1 1 51 47 0 0| 0 1248k| 45k 9476b| 0 0 | 523 760
1 2 0 0|8192b 1448k| 41k 7826b| 0 0 | 552 805
1 1 0 0|8192b 1120k| 44k 8090b| 0 0 | 470 692

Slave3:
----total-cpu-usage-----dsk/total--net/total----paging-----system--
USR sys IDL Wai hiq siq| Read writ| Recv send| In-out | int CSW
1 1 0 0|8192b 1328k|1167b 170b| 0 0 | 385 515
1 1 Wuyi 0 0|8192b 1128k| 754B 170b| 0 0 | 325 449
1 1 50 49 0 0| 0 920k| 474B 314b| 0 0 | 279 381
0 1 0 0|8192b 664k|1633b 170b| 0 0 | 226 291
1 1 0 0|8192b 1200k|1250b 170b| 0 0 | 353 475
1 1 50 48 0 0| 0 1432k|1632b 170b| 0 0 | 402 551
1 1 51 48 0 0| 16k 1752k|1045b 170b| 0 0 | 487 664
1 1 0 0|8192b 1648k| 12k 170b| 0 0 | 461 636
1 1 51 48 0 0| 0 1272k| 886B 170b| 0 0 | 380 501
1 1 0 0|8192b 1000k|1023b 170b| 0 0 | 300 400
1 1 0 0|8192b 1096k| 747B 170b| 0 0 | 332 442
1 1 0 0|8192b 1448k|1003b 170b| 0 0 | 416 557
1 1 50 48 0 0| 0 1592k|1174b 170b| 0 0 | 450 614
1 1 Wuyi 0 0|8192b 1416k|1028b 170b| 0 0 | 404 552
0 1 0 0|8192b 1128k|1031b 170b| 0 0 | 331 447
1 1 Wuyi 0 0|8192b 1160k|1185b 170b| 0 0 | 340 458
1 1 50 49 0 0| 0 1120k| 633B 170b| 0 0 | 326 453
1 0 0 0|8192b 656k|8886b 170b| 0 0 | 221 288
1 1 0 0|8192b 1128k|1619b 170b| 0 0 | 335 451

Slave1 can achieve 9M per second write Io, and slave2,3 can only reach 1M per second, IO performance is much worse, later analysis of the next storage, found that there is a great difference, but also confirmed my speculation

So the question is, how to optimize the slave of poor performance of the IO, actually very simple, modify two parameters
mysql> set global sync_binlog=20;
Query OK, 0 rows Affected (0.00 sec)

mysql> set global innodb_flush_log_at_trx_commit=2;
Query OK, 0 rows Affected (0.00 sec)

Innodb_flush_log_at_trx_commit
If Innodb_flush_log_at_trx_commit is set to 0,log buffer, it will be written to log file once per second, and the flush (brush to disk) of the log file Operation at the same time. In this mode, the operation to write to the disk is not actively triggered when the transaction commits.
If Innodb_flush_log_at_trx_commit is set to 1, MySQL writes log buffer data to log file each time the transaction commits, and flush (swipe to disk).
If Innodb_flush_log_at_trx_commit is set to 2, MySQL writes log buffer data to log file each time the transaction commits. But the flush (brushed to disk) operation does not work at the same time. In this mode, MySQL executes the flush (swipe to disk) operation once per second.
Attention:
This "flush (brush to disk) operation per second" is not guaranteed to be 100% "per second" due to a process scheduling policy issue.

Sync_binlog
The default value of Sync_binlog is 0, like the mechanism of the operating system brushing other files, MySQL does not sync to disk but relies on the operating system to flush binary logs.
When Sync_binlog =n (n>0), MySQL synchronizes its binary log binary logs to disk using the Fdatasync () function when it writes N binary log binary logs every nth time.
Attention:
If Autocommit is enabled, then each statement statement a write operation, otherwise each transaction corresponds to a write operation.
And the MySQL service is autocommit open by default

After modifying the parameters, slave2,3 can also keep up with the slave1 speed.
slave2,3:
----total-cpu-usage-----dsk/total--net/total----paging-----system--
USR sys IDL Wai hiq siq| Read writ| Recv send| In-out | int CSW
3 2 94 2 0 0| 32k 80k| 49k 10k| 0 0 |1042 658
3 2 94 1 0 1| 32k 72k| 49k 10k| 0 0 |1258 964
2 2 95 2 0 0| 32k 72k| 44k 9146b| 0 0 |1126 882
2 1 95 2 0 0| 32k 72k| 41k 8486b| 0 0 | 959 659
2 2 96 1 0 0| 32k 72k| 47k 9476b| 0 0 |1153 841
2 2 95 2 0 0| 24k 72k| 39k 8090b| 0 0 | 866 504
2 2 96 1 0 0| 24k 72k| 42k 7562b| 0 0 | 908 663
2 1 95 2 0 0| 40k 72k| 52k 10k| 0 0 |1084 685
3 1 94 2 0 1| 40k 80k| 54k 11k| 0 0 |1204 873
2 2 96 1 0 0| 16k 32k| 30k 6044b| 0 0 | 846 802
2 1 97 1 0 0| 24k 32k| 35k 7760b| 0 0 |1059 888
2 1 95 3 0 0| 32k 856k| 44k 9278b| 0 0 | 943 551
2 1 94 3 0 0| 32k 104k| 42k 8618b| 0 0 | 986 704
2 1 96 1 0 0| 24k 72k| 34k 7034b| 0 0 | 863 682
2 2 95 2 0 0| 32k 64k| 45k 8684b| 0 0 |1052 750
2 2 90 7 0 0| 24k 416k| 38k 7166b| 0 0 | 906 722
3 2 93 2 0 1| 32k 80k| 57k 10k| 0 0 |1069 829
3 2 94 1 0 0| 32k 72k| 42k 8486b| 0 0 |1076 942
2 1 96 1 0 0| 24k 72k| 37k 7496b| 0 0 | 859 575
2 2 94 2 0 1| 32k 64k| 43k 8684b| 0 0 |1138 1011
3 2 94 1 0 0| 32k 72k| 42k 9014b| 0 0 |1099 782
2 3 94 2 0 0| 32k 72k| 50k 10k| 0 0 |1332 1359
2 2 95 2 0 0| 24k 72k| 34k 6902b| 0 0 | 921 799
2 2 94 2 0 0| 40k 72k| 55k 11k| 0 0 |1318 1016
1 2 96 2 0 0| 32k 80k| 41k 8882b| 0 0 |1020 719
And I observed that the number of writes in slave2,3 decreased by two orders of magnitude, from 1M to 70k

Optimize synchronization speed for MySQL slave