The Bytebuffermessageset of Kafka source code analysis

Here is an analysis of the wrapper class of message Bytebuffermessageset class

The source code of the Bytebuffermessageset class is under the message directory of the sources directory. This class mainly encapsulates the Message,messageset, Messageandoffset objects such as the class. The log class is basically manipulating objects with objects of this class.

Here's a look at the specific code for the class. First, the initialization section.

classBytebuffermessageset (Val buffer:bytebuffer)extendsMessageset with Logging {//here as ingress into the Bytebuffer type bufferPrivatevar shallowvalidbytecount = 1//sets the global validation byte statistic variable.//Here are a few different constructors. But all are based on the original constructor. The Create method of the Bbms object is called here for creating a new buffer. def This(Compressioncodec:compressioncodec, messages:message*) {//parameter is a compression flag and message This(Bytebuffermessageset.create (NewAtomiclong (0), Compressioncodec, messages:_*)}//Ibid. Only the parameters are different. def This(Compressioncodec:compressioncodec, Offsetcounter:atomiclong, messages:message*) {     This(Bytebuffermessageset.create (Offsetcounter, Compressioncodec, messages:_*) }//The simplest message constructor. def This(messages:message*) {     This(Nocompressioncodec,NewAtomiclong (0), messages: _*)  }

The above is a few different constructors sections. The Bytebuffermessageset.create function code that the constructor applies to is as follows.

The parameter of object Bytebuffermessageset {//create function is offset, compressed, message. Returns the Bytebuffer type. PrivateDEF create (Offsetcounter:atomiclong, Compressioncodec:compressioncodec, messages:message*): Bytebuffer = {    if(Messages.size = = 0{//The message is empty, return an empty buffer MessageSet.Empty.buffer}Else if(Compressioncodec = =Nocompressioncodec)      {//without compression. Assign a new messages size buffer. Val Buffer=bytebuffer.allocate (Messageset.messagesetsize (Messages)) for(Message <-messages) writemessage (buffer, message, offsetcounter.getandincrement)//Here writes the message contents to buffer.      Buffer.rewind ()//sets the pointer to the start position. Buffer//return buffer}Else{//Here is the processing action to enable compression. Val Bytearraystream=NewBytearrayoutputstream (Messageset.messagesetsize (Messages))//Get a byte stream object of messages size Val output=NewDataOutputStream (Compressionfactory (Compressioncodec, Bytearraystream))//Create write object. var offset= -1lTry {         for(Message <-messages)          {//message compression is written to the byte stream object. Offset=offsetcounter.getandincrement Output.writelong (offset) output.writeint (message.size) out Put.write (Message.buffer.array, Message.buffer.arrayOffset, Message.buffer.limit)}}finally{output.close ()} Val bytes=Bytearraystream.tobytearray//Turn into byte Val message=NewMessage (bytes, compressioncodec)//Create a compressed Message object. Val Buffer= Bytebuffer.allocate (Message.size +messageset.logoverhead)/Assign a buffer writemessage (buffer, message, offset)//write the compressed message to buffer. Buffer.rewind () Buffer}} def decompress (message:message): Bytebuffermessageset={val Outputstream:bytearrayoutputstream=NewBytearrayoutputstream Val Inputstream:inputstream=NewBytebufferbackedinputstream (message.payload) Val Intermediatebuffer=NewArray[byte] (1024) Val Compressed=compressionfactory (Message.compressioncodec, InputStream)Try{stream.continually (Compressed.read (Intermediatebuffer)). TakeWhile (_> 0). foreach {Dataread = =Outputstream.write (Intermediatebuffer,0, Dataread)} } finally{compressed.close ()} Val outputbuffer=bytebuffer.allocate (outputstream.size) outputbuffer.put (outputstream.tobytearray) OutputBuffer.rewindNewBytebuffermessageset (OutputBuffer)}//write message to bufferPrivate[Kafka] def writemessage (Buffer:bytebuffer, Message:message, Offset:long) {Buffer.putlong (offset)//write offset first    is the size, and finally the message itself. Buffer.putint (message.size) buffer.put (Message.buffer) Message.buffer.rewind ()}}

The above is about the initialization part of the work. Here's a look at the specific function. This function is used as the core function. The messages map to the corresponding Messageandoffset is implemented here. Several functions that are called externally are also dependent on the function implementation.

Override def Iterator:iterator[messageandoffset] =Internaliterator ()/**iterator over compressed messages without decompressing*/def Shallowiterator:iterator[messageandoffset]= Internaliterator (true)  /**When flag Isshallow are set to BES true, we do a shallow iteration:just traverse the first level of messages. **/  Privatedef internaliterator (Isshallow:boolean =false): Iterator[messageandoffset] ={//Can see the return is an iterative object, the object class wants to be MessageandoffsetNewIteratortemplate[messageandoffset] {//This is an abstract class. In this place, an instance of an anonymous class is returned. var topiter=Buffer.slice ()//copy a buffer var inneriter:iterator[messageandoffset]=NULLdef innerdone (): Boolean= (Inneriter = =NULL|| !inneriter.hasnext) def Makenextouter:messageandoffset={//There is no message mapped to Messageandoffset//if there isn ' t at least an offset and size, we 're done        if(Topiter.remaining < 12)            returnalldone () Val offset=Topiter.getlong () Val size=Topiter.getint ()if(Size <message.minheadersize)Throw NewInvalidmessageexception ("Message found with corrupt size (" + size + ")")                //we have an incomplete message        if(Topiter.remaining <size)returnalldone ()//above these are mainly to check what. Not in detail. The primary mapping is completed below. //read the current message and check correctnessVal message =Topiter.slice () message.limit (size)//intercepts the first message topiter.position with a size (topiter.position+size)//move the pointer forward. Val Newmessage=Newmessage (message)//build new Messageif(isshallow) {NewMessageandoffset (newmessage, offset)//do mapping back. } Else{//The rest is to determine if there is a compression condition. Newmessage.compressioncodec Match { CaseNocompressioncodec =Inneriter=NULL              NewMessageandoffset (newmessage, offset) Case_ =Inneriter=bytebuffermessageset.decompress (newmessage). Internaliterator ()//compression is handled by the method in the object. if(!inneriter.hasnext) Inneriter=NULLMakenext ()}}      }//Call the mapping function in this function. Finally, this function has the next function call. Override Def Makenext (): Messageandoffset= {        if(Isshallow) {makenextouter}Else {          if(Innerdone ()) MakenextouterElseInneriter.next} }}}

This is the process of a core function. Look at the definition of the corresponding abstract class iteratortemplate.

Abstract classITERATORTEMPLATE[T]extendsiterator[t] with java.util.iterator[t] {Privatevar state:state =Not_readyPrivatevar Nextitem =NULL. Asinstanceof[t] def next (): T= {    if(!Hasnext ())Throw Newnosuchelementexception () state=Not_readyif(Nextitem = =NULL)      Throw NewIllegalStateException ("Expected item but none found.")) Nextitem} def Peek (): T= {    if(!Hasnext ())Throw Newnosuchelementexception () Nextitem} def hasnext (): Boolean= {    if(state = =FAILED)Throw NewIllegalStateException ("Iterator is in failed state") state Match { CaseDone =false       CaseReady =true       Case_ =Maybecomputenext ()}} protecteddef makenext (): T def maybecomputenext (): Boolean={ state=FAILED Nextitem=Makenext ()if(state = =Done ) {      false    } Else{ state= Readytrue    }  }    protectedDef alldone (): T ={ state= DoneNULL. Asinstanceof[t]} def remove=Throw NewUnsupportedoperationexception ("Removal not supported")  protecteddef resetstate () { state=Not_ready}}

The definition is simple and straightforward.

The Bytebuffermessageset of Kafka source code analysis