MySQL Binlog parsing summary

1,dump Protocol:

Create a socket based on the ip+port of the database, if the creation succeeds, the link is established successfully, then the dump protocol is used to subscribe Binlog

After the link is established successfully, the server will actively send the following greeting message to the client greeting (it can be understood that after the Java conversion, it is a Java object),

You can see the information in greeting in the following code:

This.context.setServerStatus (Greeting.getserverstatus ());//
This.context.setServerVersion (Greeting.getserverversion (). toString ());
This.context.setServerCollation (Greeting.getservercollation ());
This.context.setServerCapabilities (Greeting.getservercapabilities ());
This.context.setThreadId (Greeting.getthreadid ());
This.context.setProtocolVersion (Greeting.getprotocolversion ());
This.context.setScramble (Greeting.getscramble1 (). ToString () + greeting.getscramble2 (). ToString ());

Then the following:
Will Ctx=this.context;

Final Xserializer s = new Xserializer (64);
S.writeint (Buildclientcapabilities (), 4);
S.writeint (This.maximumpacketlength, 4);
S.writeint (this.clientcollation > 0 this.clientCollation:ctx.getServerCollation (), 1);
S.writebytes ((byte) 0, 23); Fixed, all 0
S.writenullterminatedstring (Stringcolumn.valueof (This.user.getBytes (this.encoding)));
S.writeint (20, 1); The length of the SHA1 encrypted password
S.writebytes (Mysqlutils.password41orlater (This.password.getBytes (this.encoding), ctx.getscramble (). GetBytes ( (this.encoding)));
if (This.initialschema! = null) s.writenullterminatedstring (stringcolumn.valueof (This.initialSchema.getBytes ( (this.encoding)));

Transport can be understood to be a socket after the wrapping of something
Final Rawpacket request = new Rawpacket ();
Request.setsequence (1);
Request.setpacketbody (S.tobytearray ());
Request.setlength (Request.getpacketbody (). length);
Transport.getoutputstream (). Writepacket (Request);
Transport.getoutputstream (). Flush ();

You can then get the corresponding MySQL in the following code:

Final Packet response = Transport.getinputstream (). Readpacket ();
if (Response.getpacketbody () [0] = = Errorpacket.packet_marker) {
   Final Errorpacket error = errorpacket.valueof (response);
   Logger.info ("Login failed, User: {}, Error: {}", This.user, error);
   throw new Transportexception (error);
} else if (Response.getpacketbody () [0] = = Okpacket.packet_marker) {
   Final Okpacket OK = okpacket.valueof (response);
   Logger.info ("Login successfully, User: {}, Detail: {}", This.user, OK);
} else {
   Logger.warn ("Login failed, Unknown packet:", response);
   throw new RuntimeException ("Assertion failed, Invalid packet:" + response);
}

2,dump Message Format:

The table below is from the MySQL website link: http://dev.mysql.com/doc/internals/en/mysql-packet.html

type	name	description
Int<3>	payload_length	Length of the payload. The number of bytes in the packet beyond the initial 4 bytes the packet header.
Int<1>	sequence_id	Sequence ID
String<var>	payload	[Len=payload_length] payload of the packet

The message supports the super-large packet as follows:

When a packet is too large (more than a 1<<24-1 byte ~= MB), the transmission needs to be cut for packet, see here

Note that when generating binlog on a, it is possible to accommodate packet greater than 16MB, that is, there is an oversized event in the original Binlog, which needs to be restricted during transmission

Cut packet Nothing special, only need to note the package format, a 20MB event of the transmission packet format for example (here with 16MB ease of description, should be 1<<24-1 byte):

    1        4字节 packet header        1字节 值为[00], 是binlog event的特征标志 16MB-1字节 为第一段数据 packet 2 4字节 packet header 20MB-16MB+1字节 为第二段数据

It is important to note that the subsequent packet does not have a [00] feature bit. The size of the package is calculated to remove all bytes from the first 4 bytes

The corresponding code for the above explanation is:

Parse Packet
Final int packetlength = Is.readint (3);
Final int packetsequence = Is.readint (1);//order after large packet cut
Is.setreadlimit (packetlength); Ensure the packet boundary

//
Final int packetmarker = Is.readint (1);//Feature bit
if (packetmarker! = Okpacket.packet_marker) {//0x00
   if ((byte) Packetmarker = = Errorpacket.packet_marker) {
      Final Errorpacket packet = errorpacket.valueof (Packetlength, Packetsequence, Packetmarker, is);
      throw new RuntimeException (packet.tostring ());
   } else if ((byte) Packetmarker = = Eofpacket.packet_marker) {
      Final Eofpacket packet = eofpacket.valueof (Packetlength, Packetsequence, Packetmarker, is);
      throw new RuntimeException (packet.tostring ());
   } else {
      throw new RuntimeException ("Assertion failed, invalid packet marker:" + packetmarker);
   }
}

Then directly parse the MySQL Binlog, the header gets as follows:

Parse the event header
Final Binlogeventv4headerimpl Header = new Binlogeventv4headerimpl ();
Header.settimestamp (Is.readlong (4) * 1000L);//timestamp
Header.seteventtype (Is.readint (1));//type_code
Header.setserverid (Is.readlong (4));//server_id
Header.seteventlength (Is.readint (4));//event_length
Header.setnextposition (Is.readlong (4));//next_position
Header.setflags (Is.readint (2));//flags
Header.setbinlogfilename (This.binlogfilename);
Header.settimestampofreceipt (System.currenttimemillis ());

Body parsing different event format, like header value



Note: Code from open source project open Replicator

MySQL Binlog parsing summary