The logsegment of Kafka source code analysis

This analysis Kafka logsegment source code

By analyzing the Logmanager,log source code in one step, you will find that The final log operation is implemented on the logsegment. Logsegment is responsible for the read and write recovery of the Shard Delete, and other actions are implemented here. The Logsegment code is also under the source code directory log.

Logsegment is the smallest unit of operation of a log shard. Direct action and messages. Responsible for reading and writing of entity messages and so on.

Logsegment is actually the proxy class for the Filemessageset class. All final processing in the logsegment is implemented in the Filemessageset class. The final operation of the Filemessageset class is based on the fact that the message On the basis of the body class. Implement message read and write by manipulating FileChannel objects.

Here's a look at some of the main function methods.

Initialization section

classLogsegment (Val log:filemessageset,//actual construction is this.                 Val Index:offsetindex, Val Baseoffset:long, Val indexintervalbytes:int, Val Rolljitterms:long, Time:time)extendsLogging {var created=Time.milliseconds/*The number of bytes since we last added a entry in the offset index*/  Privatevar bytessincelastindexentry = 0//The construct called in log is this. You can see that the index and logfile are created by topicandpartition paths and Startoffset. def This(Dir:file, Startoffset:long, Indexintervalbytes:int, Maxindexsize:int, Rolljitterms:long, time:time) = This(NewFilemessageset (file =Log.logfilename (dir, Startoffset)),NewOffsetindex (file = Log.indexfilename (dir, startoffset), Baseoffset = startoffset, maxindexsize =maxindexsize), Startoffset, Indexintervalbytes, rolljitterms, time)

Add Message function Append

def append (Offset:long, messages:bytebuffermessageset) {if(Messages.sizeinbytes > 0{//To determine that the message is not empty. Trace ("Inserting%d bytes at offset%d at position%d". Format (messages.sizeinbytes, offset, log.sizeinbytes ()))//Append an entry to the index (if needed)      if(Bytessincelastindexentry >indexintervalbytes) {index.append (offset, log.sizeinbytes ()) This. Bytessincelastindexentry = 0      }      //Append the Messageslog.append (Messages)//Call the Append method of the Filemessageset class to write a message. The Bytebuffermessageset class method is actually called to manipulate the message entity.  This. bytessincelastindexentry + =Messages.sizeinbytes}}

　　Flush function for flushing messages to disk

DEF flush () {    LogFlushStats.logFlushTimer.time {      log.flush ()   /// You can see the method that calls the Filemessageset class. The final Filemessageset.flush method calls the Channel.force method to flush the storage device.      Index.flush ()//Ibid.    }  }

Read function for reading messages

def read (Startoffset:long, Maxoffset:option[long], maxsize:int): Fetchdatainfo = {    if(MaxSize < 0)      Throw NewIllegalArgumentException ("Invalid max size for log read (%d)". Format (maxSize)) Val logsize= Log.sizeinbytes//need to save a consistent copyVal startposition =Translateoffset (Startoffset)//Gets the location of the read point corresponding to offset. //If the start position is already off the end of the log, return null    if(StartPosition = =NULL)//No read point location returns nullreturn NULLVal Offsetmetadata=NewLogoffsetmetadata (Startoffset, This. Baseoffset, startposition.position)//Definition Offsetmetadata//If the size is zero, still return a log segment but with zero size    if(MaxSize = = 0)//maximum read size is 0. Returns an empty message. returnFetchdatainfo (Offsetmetadata, Messageset.empty)//calculate the length of the message set to read based in whether or not they gave us a maxoffsetVal length =//Calculates the total length of the message for maximum reads. Maxoffset Match { CaseNone =//Not set Maxoffset the maxsize is used.//no max offset, just use the max size they gave unmolestedmaxSize CaseSome (offset) ={//If Maxoffset is set, the corresponding message length is calculated. //there is a max offset, translate it to a file position and use this to calculate the max read size          if(Offset <Startoffset)//maxoffset less than Startoffset returns an exceptionThrow NewIllegalArgumentException ("Attempt to read with a maximum offset (%d) less than the start offset (%d).". Format (offset, startoffset)) Val mapping=translateoffset (offset, startposition.position)//Gets the relative Maxoffset read point. Val endposition=if(Mapping = =NULL) LogSize//The max offset is off the end of the log and use the end of the file            Elsemapping.position min (endposition-startposition.position, MaxSize)//Subtract the starting read point with the Maxoffset read point. Gets the length of the data that needs to be read. Returns maxSize}} fetch if the length is greater than maxSize Datainfo (Offsetmetadata, Log.read (startposition.position, length))// Use Filemessageset.read to read the corresponding length of data to return the encapsulated object of Fetchdatainfo.}

Reads the function by mapping offset to the read length. To read multiple offset.

Private [Log] def translateoffset (offset:long, startingfileposition:int = 0): Offsetposition = {  // The function used to map offset to the position of the    reading pointer. = Index.lookup (offset)//Gets the corresponding pointer object by looking for index.    Log.searchfor (offset, max (mapping.position, startingfileposition))//Gets the corresponding pointer position through Filemessageset.  }

The Recover function. Kafka the last proxy function called by each layer that was used to start the check.

def recover (maxmessagesize:int): Int ={index.truncate () index.resize (index.maxindexsize) var validbytes= 0var lastindexentry= 0Val iter=log.iterator (maxmessagesize)Try {       while(Iter.hasnext) {Val entry=iter.next entry.message.ensureValid ()if(Validbytes-lastindexentry >indexintervalbytes) {          //we need to decompress the message, if required, to get the offset of the first uncompressed messageVal Startoffset =Entry.message.compressionCodec Match { CaseNocompressioncodec =Entry.offset Case_ =bytebuffermessageset.decompress (entry.message). Head.offset} index.append (STARTOFFSE T, validbytes) Lastindexentry=Validbytes} validbytes+=messageset.entrysize (entry.message)}} Catch {       CaseE:invalidmessageexception =Logger.warn ("Found Invalid messages in log segment%s at byte offset%d:%s.". Format (Log.file.getAbsolutePath, Validbytes, E.getmessage))} Val truncated= Log.sizeinbytes-validbytes Log.truncateto (validbytes) index.trimtovalidsize () truncated}

Shard Delete function

def delete () {    = log.delete ()   //finally deletes the file and closes the memory array. In Filemessageset.     = Index.delete ()//Ibid.     if (!deletedlog && log.file.exists)       Throw New Kafkastorageexception ("Delete of log" + Log.file.getName + "failed." )    if(!deletedindex && index.file.exists)      throw New Kafkastorageexception ("Delete of index" + Index.file.getName + "failed." )  }

Here Logsegment the main functions are analyzed.

The logsegment of Kafka source code analysis