Solve the BUG of failed Binlog clearing during tunstenapi Synchronization

After you modify tunstenapi to parse logs, the log is cleared before switching to the next log. The purpose of this operation is to prevent RelayLog from synchronizing data to mysql.

After you modify the tunsten API to parse logs, the log is cleared before switching to the next log. The purpose of this operation is to prevent RelayLog from synchronizing data to mysql.

After you modify the tunsten API to parse logs, the log is cleared before switching to the next log. The purpose of this operation is to prevent excessive disk space occupation caused by RelayLog synchronization of mysql master logs. This operation adds clearFile operations in the BinlogPosition reset method.

One problem found during application implementation: the first resolved Binlog cannot be deleted, and occasionally a binlog cannot be cleared. (For uncleared log files, there will be records and will be deleted again when the file is cleared next time. During the operation, if we have enough binlog logs, after multiple retries, the first binlog can be deleted successfully)

Through the above analysis, it is estimated that the first Binlog log should have a special logic, which causes the file handle to be retained in the memory before the file is deleted, in addition, it is estimated that this handle will remain in the JVM for a period of time. When GC occurs, this handle can be recycled normally.

The same is found in the MysqlExtractor parsing source code debugging. In the processEvent method, when the InputStream stream opened is not empty and the Binlog Position is 0, a new file is openFile, there is no problem in opening the new file logic, but its processing code is:

It is possible that InputStream is not empty,

However, it is directly used before it is disabled: FS = new FileInputStream (file )//

This type of problem is not advisable in file processing. If the original file is not closed, the file handle also occupies space in JVM, and the file cannot be cleared by other applications. (Until GC)

(In the process, the position of the first Binlog processing starts from 0! Because the API call does not consider parsing from the middle of the log and saving the status during the parsing process)

The above BUG solves the problem of file stream operations. Before the above operation, the stream is not empty, and the judgment of the stream is disabled, the file can be deleted normally.

The last two notes are typical and profound bugs encountered during the synchronization process based on the tunsten API during mysql binlog processing.

(The latest project is to deal with mysql binlog to discover specific SQL changes and develop data consistency for another system. This is a small attempt. Now the development is coming to an end, and both real-time and feasibility are acceptable. Starting next week, we will consider updating the database to the redis cache, (the first stage of the system is to implement redis-based distributed cache by itself), but it has some limitations and is highly coupled with the DAO of the application itself. Next, we will record the relevant research and related issue records in the development process. We will also record the first team segment and redis-based distributed Cache architecture for future viewing .)