Mysql replication error: Relay_Log_Pos unchanged

Recently, a Mysql SLAVE of the company experienced a delay of seconds. When we found the delay two days ago, we observed the slave status. Everything was normal and thought that the recent business adjustment caused too many write operations, the slave machine has a high latency, so you don't have to pay too much attention to it. After coming back from the New Year's Day holiday, there was a delay.

The specific situation is described as follows: the IO and SQL threads run normally, and the master log copy is normal. The only abnormal problem is that Relay_Log_Pos remains unchanged and the slave status does not report an error. It is caused by a table locked for a long time by a statement. However, processlist finds that no statement is running in the database. In order not to affect the business operation, we migrate all the read operations to another one and analyze the cause of this machine. The slave status is as follows:

After several set global SQL _SLAVE_SKIP_COUNTER = 1 Operations (skip the current execution position), we found that the problem still persists. So we want to find out the cause of the jam. Based on the given location, our team relay_log analyzed

Mysqlbinlog relaylog.008077 -- start-position = 223496095-f-d xxx | more results are as follows:

We found that every time the stuck position is in front of the unkown event, it will be stuck when we encounter this problem every time we skip it. The reason for this time in the relay log is unknown, probably because we have adjusted the table structure partition. Finally, after the set global SQL _SLAVE_SKIP_COUNTER = 1 operation is executed multiple times, the system will no longer get stuck.