On the correspondence between timestamp and offset in Kafka

on the correspondence between timestamp and offset in Kafka

@ (KAFKA) [Storm, KAFKA, big Data]

On the correspondence between timestamp and offset in Kafka gets the case of a single partition and gets the message from all the partitions at the same time how to specify the processing method when the timing occurs updateoffsetexception related source skimming 1 ingress 2 processing logic 1 establish the correspondence between offset and timestamp and save it to the data 2 find the nearest last one to satisfy the timestamp Target_timestamp index 3 to find the offset array 3 considerations that satisfy the condition

get a single partition case

Kafka the offset of each log by offset, see the Kafka file storage mechanism. However, when the user wants to read the previous information, he is not able to know the corresponding offset of these messages, the user can only specify the time, for example, I started from yesterday's 12 point to read the message.

This is a question of how to translate user-defined time into the intra-cluster offset.

Just revisit the physical storage mechanism of Kafka: Each topic is divided into multiple partitions, and one partition corresponds to a directory on the disk, and there are multiple files in the directory, such as:

00000000000000000000.index  00000000000000000000.log  00000000000001145974.index  00000000000001145974.log

As you can see, each segment file actually has 2 parts, an index file, and a log file. The filename is the offset of the first message within this file. The log file records the actual message content. While index logs the log file, when you need to view a message, if you specify offset, it is easy to navigate to the exact location in the log file. See the article above.

But as I said, users do not know offset, but only know the time, so they need to convert.

Kafka used a very straightforward method: Place offset in the file name in a map with the last modification time of the document, and then look for it. The detailed steps are as follows:
(1) put the filename and the last time of the file into a map, time using a 13-bit UNIX timestamp
(2) When the user specifies a time t0, the last time in the map is T1 earlier than T0, and then returns the file name, which is the first offset of the file.
(3) This returns only one partition of offset, which in fact needs to return all partitions of offset, so the above steps are taken for all partitions.
(4) Use the messages you have taken to start consuming messages.

As an example:

 w-r--r--1 Hadoop hadoop 1073181076 8?? 10:20 00000000000066427499.log-rw-r--r--1 Hadoop hadoop 14832 8?? 10:20 00000000000066427499.index-rw-r--r--1 Hadoop hadoop 1073187364 8?? 10:40 00000000000067642947.log-rw-r--r--1 Hadoop hadoop 14872 8?? 10:40 00000000000067642947.index-rw-r--r--1 Hadoop hadoop 1073486959 8?? 11:04 00000000000068857698.log-rw-r--r--1 Hadoop hadoop 14928 8?? 11:04 00000000000068857698.index-rw-r--r--1 Hadoop hadoop 1073511817 8?? 11:25 00000000000070069880.log-rw-r--r--1 Hadoop hadoop 14920 8?? 11:25 00000000000070069880.index-rw-r--r--1 Hadoop hadoop 10485760 8?? 11:28 00000000000071279203.index-rw-r--r--1 Hadoop hadoop 148277228 8?? 11:28 00000000000071279203.log 

We have several of the above documents
(1) When I need to consume data starting from August 11 11:00, it will return the file name with the last modified time earlier than August 11 11:00, and also the file modified at time 10:40. Offset is 67642947. In fact, due to its last modification time at 10:40, the data we need is not possible in it, it directly returns 11:40 of the file can be, but perhaps for more insurance considerations, it returned the previous file.
(2) Other similar, when I consume 11:20 of the data, the returned offset is 68857698.
(3) and when the data I consume is earlier than 10:20, the returned offset is empty, and if the offset is saved by an array, the java.lang.ArrayIndexOutOfBoundsException exception occurs when the first offset is fetched. As in the code in Simpleconsumer in the Kafka Programming Guide:

    Long[] Offsets = response.offsets (topic, partition);
    return offsets[0];

Of course, it can also be handled reasonably, when returned to empty, directly return the earliest offset.

(4) When the consumption data is later than the last time, the latest message is returned.

Attention:
(1) There is no burden on the Kafka cluster itself, the Kafka message does not need to record the time point of this field, only when the need to locate the time, only to temporarily build a map, and then read offset and time into the map.
(2) Redundancy of many messages. The granularity of this method is very coarse, the file as the granularity, so the redundant message data and the size of the file has a relationship, the default is 1G. If this topic data is very small, then this 1G data can be a few days ago data.
(3) There are 2 special points of time:
The timestamp to be searched is 1 or-2, special handling

  Case offsetrequest.latesttime =  //offsetrequest.latesttime =-1    
    startIndex = offsettimearray.length-1 Case  
  Offsetrequest.earliesttime =//Offsetrequest.earliesttime =-2
    startIndex =0

When you get messages from all partitions at the same time

1, when the message is read from multiple partitions at the same time, as long as there is one partition, all its files are modified later than the time you specified, it will be an error, because this partition returns the offset is empty, unless you do a reasonable deal.

2. Storm!!!
When the storm0.9x version encounters the above problem, the same error occurs, with the following exception

Storm.kafka.UpdateOffsetException

Starting with the 0.10 version, the consumer message was changed from the earliest time.

3, there is a problem, how to distribute the message evenly but in each partition. For example, in one of our topic, one of the partitions already has 60G data, and the other partition is less than 2G, if the specified time, because the small partition of the modification time is certainly in the near future, so when specifying a more previous point in time will be wrong. And even if there is no error, messages that are never returned from the partition can be very long.

How to distribute data evenly across partitions, refer to the partitioner introduction to the Kafka Programming Guide.

As long as this problem occurs because the data does not exist, it is possible that:

(1) The data is really lost.

(2) Data skew serious conclusion How to specify time

If you need to specify that you want to start processing logs from a point in time:

(1) Just specify that time, do not need to advance, so the returned message must be at this point in time.

(2) If this point of time is a busy period, the message time it returns may be just a short period of time before this point in time.

(3) If this point of time is an idle time, it may return log time for a long period of time.

(4) But whether it is busy time or idle time, it is read more than one log file, so the number of redundant logs is the same.

As an example:

If you need to process the logs after 2015-08-15 15:00:00,

(1) Specify this time directly, do not need to specify the time before it, such as 1:00, 2:00, and so on, because the log time returned is never before 3:00. At the same time, 3:00 access to Kafka data may be pre-3:00 data, will never be 3:00 after the data, so there is no need to consider specifying advance time.

(2) Since this time log is generally more, the log it returns may have started around 2:30. Conversely, if it is 3:00, because this time point log is less, it returns the log can be 2, 3 hours ago. How to handle the occurrence of updateoffsetexception

(1) If the small project, because the amount of data is not much, it is recommended to start processing, and notify SA check.

(2) If a large project, usually there is data skew, notify SA check data situation. related source skimming 1. Entrance

Kafka Server handles the entry of requests sent by the Client in
Folder: Core/src/main/scala/kafka/server
Class: Kafka.server.KafkaApis
Method: Handle
Functions for handling offset requests: Handleoffsetrequest 2, processing logic

The

processing logic is mainly divided into four steps
Get partition
Get offset from partition
High water mark processing (this is too little data)
exception handling
Because the request contains multiple queries Partition of the offset request. So it will eventually return a map that holds the offset
for each partition, which mainly describes the logic to get offset from a partition, code location
Kafka.log.log#getoffsetsbefore (timestamp, maxnumoffsets)
Get offset from a partition (1) establishes the correspondence between offset and timestamp and saves it to the data

Each partition is made up of multiple segment file. Gets the segment list in the current partition
val segsarray = Segments.view

//create array
var offsettimearray:array[(long, long)] = Null
if (segsArray.last.size >0)
  Offsettimearray =newarray[(long, Long)] (segsarray.length + 1)
Else
  Offsettimearray =newarray[(long, Long)] (segsarray.length)

//Add the corresponding relationship of offset and timestamp to the array for
(i <- 0until segsarray.length)
  //Each element in the data is a two-tuple, (segment file's starting Offset,segment file's last modified time)
  Offsettimearray (i) = (Segsarray (i). Start, Segsarray (i). messageSet.file.lastModified)
if (segsArray.last.size >0)
  // If the most recent segment file is not empty, it will (the nearest offset, current between) also be added to the array
  offsettimearray (segsarray.length) = (Logendoffset, Time.milliseconds)
Through this logic, the obtained data Offsettimearray, each element is a two-tuple, two-tuple content is (offset, timestamp)

(2) Find the latest last one to meet timestamp < Target_timestamp index

var startIndex =-1
timestamp match {
  //timestamp to be found is-1 or-2, special handling
  caseoffsetrequest.latesttime =>
  //offsetrequest.latesttime =-1
    startIndex = offsettimearray.length-1
  caseoffsetrequest.earliesttime = >//Offsetrequest.earliesttime =-2
    startIndex =0
  case_ =
    var isfound =false
    Debug ("Offset Time Array = "+ Offsettimearray.foreach (o + ="%d,%d ". Format (o._1, o._2)))
    StartIndex = offsettimearray.length-1 //  Reverse the last element to find while
    (StartIndex >=0&&!isfound) {    //Find the satisfying condition or
      if (Offsettimearray ( StartIndex). _2 <= timestamp)  //Offsettimearray Each element is two tuples, the second position is timestamp
        isfound =true
      Else
        startIndex-=1
    }
}

Through this logic, we actually find the "starting offset of the most recently modified segment file with the last modified time earlier than the target timestamp"
But the logic to get offset does not end, and there is still processing (3) to find an array of offset that satisfies the condition

The function is actually to find a set of offset, not an offset. The second parameter, maxnumoffsets, specifies the maximum number of offset that satisfies the condition.

Gets the length of the logical
//returned data for a set of offset = min (maxnumoffsets, StartIndex + 1), StartIndex is the logical 2 found in the index
val retsize = Maxnumoffsets.min (StartIndex + 1)
val ret = Newarray[long] (retsize)

//One by one adds the eligible offset to the returned data for
(J < -0until retsize) {
  ret (j) = Offsettimearray (StartIndex). _1
  startIndex-=1
}

//descending sort returned. The larger the offset, the more new the data.
//Ensure that the returned SEQ are in descending order of offsets
Ret.toSeq.sortBy (-_)

Finally return to this array 3, precautions

The actual offset found is not the first offset starting from the target timestamp. Need to be aware
When timestamp is less than the most recent modification time of the oldest data file, the return value is an empty array. May cause problems when they are used.
When you adjust the configuration of the segment file split policy, you need to be aware of the possible impact.