The faulty machine is restarted after being repaired, and the binlog of the master database is frantically pulled, resulting in a solution to network problems _ MySQL

This article mainly records a simple and typical fault. The cause of the problem is very simple, and the problem is also very simple. you must note that it will affect the master database if you don't care about it. Problem description:

A week ago, a mysql server was shut down due to hardware faults. We submitted an application to the student responsible for this part. they are responsible for reporting to the student responsible for repairing this server. After the server is repaired today, they start it up. The four mysql instances on the server are automatically started after the instance is started, and the binlog of the master database is pulled. Because the server has been down for a long time, many logs are lost, and the binlog of the master database is pulled, the network of the master database becomes faulty.

Symptom:
First of all, we didn't realize it was caused by a bad server restarting and pulling the binlog of the master database, because we didn't know what the server was like, just one week ago, we have repaired one server. We have no idea about the specific situation, whether it has been repaired, or whether it has been turned on.

In this case, I suddenly heard the network staff say that the network traffic of a mysql machine is too large, resulting in a very slow business experience, lasting for 17 minutes in total. In fact, there is no clue.

Troubleshooting:

No problems were found in processlist, full logs, and slow logs.

Check monitoring and find that the server's read IO suddenly increased during that time.

By viewing the processlist history, we found that for a period of time, the user status of master-slave replication was waiting for net, and its IP address found that the server was a server that crashed one week ago.

Conclusion:

There are four instances on this server. after the server is started, the mysql instance starts automatically and starts to pull the binlog to the master database. The Daily binlog volume of each master database is about 6 GB, four instances have a binlog of about 160 GB in a week.

Problem:

1. When will the faulty server be repaired or when it will be started? we are not able to control it, nor are we concerned about it.

2. this case is actually a very simple and typical case that may cause an impact or failure. we are not aware of this phenomenon in advance. although we know that this is a very easy problem, however, in our case, there is no such awareness. Therefore, this event occurs.

3. lack of effective monitoring on network traffic

Solution:

1. stop all servers from starting mysql automatically. after the server is started, manually start the instance and stop slave. (In this way, if there are a lot of servers, it may be too troublesome. for the time being, it will be recorded first, which is always more influential)

2. be aware of the problem and include it in the knowledge library or work manual for problem avoidance.

The above is the solution to the network problem caused by the binlog of the master database after the faulty machine is repaired and restarted. _ MySQL content. For more information, see PHP Chinese network (www.php1.cn )!