"Original" KAKFA common package source code analysis

The first day to look at the common package code startled, so many scala files! After a closer look at most of the Kafka custom exception class, can be changed to be called Kafka.exceptions package. Since the names of the exception classes are often defined intuitively, I will not repeat them in one by one. Now let's talk about the other code in the common package.

First, Appinfo.scala

This is an object that implements the Kafkametricsgroup trait. The latter can be thought of as a factory class that creates various measures. The main use of the Yammer metrics Framework is to create a variety of metrics, such as Guage,meter,timer and histogram. Instead, only one method is defined in the AppInfo object: Registerinfo

The method does not return a value, it first determines whether it has been registered (isregistered = = true): If it is registered to return directly. The class is then judged whether it comes from the jar package, and if not, it is returned directly. The last program creates a manifest object and reads the version information from it and uses that version information to create a guage.

Second, Clientidandbroker.scala

Found in the Kafka source code there are many similar to AANDB or ab such naming, basically is to create a class, receive 2 parameters A and b meaning. This scala file is no exception, with two case classes: Clientidandbroker and clientidallbrokers. Basically the former is to create a class to hold a ClientID and a broker, while the latter is given a clientid output clientid-allbrokers.

Third, Clientidtopic.scala

Exactly the same as Clientidandbroker.scala, just this time saving (clientId, topic) information-the tuple pair is very useful because they are used in a lot of stats object creation later.

Four, Config.scala

The code is not complicated, it is just a simple trait, there is only one method without return value: validatechars--is also very simple to verify that there are no illegal characters in the given attribute value. Currently, only letters, numbers, periods (.), underscores (_), and horizontal lines (-) are allowed. Other characters are treated as illegal characters, and once found storing illegal characters immediately throws an exception. The config trait is implemented by producer and consumer with the corresponding CONFIG.

Five, Errormapping.scala

Make a sequence number for each exception in the common package, add it to a map, and provide many ways to retrieve it-that is, to return an exception instance based on the ordinal or to return the exception ordinal based on the exception

Six, Offsetmetadataanderror.scala

Two groups of associated objects are available in the Scala file: Offsetandmetadata and Offsetmetadataanderror. Let's start with the first group.

The Offsetandmetadata object provides three constant fields: Invalidoffset (-1), Nometadata (""), and Invalidtime (-1). Used primarily for initialization of offsetandmetadata fields when an error occurs. Thus, the class constructor of the same name also receives three parameters: the displacement information of the long type, the metadata information of the string representation, and a timestamp. This class provides a tostring method to format the output of this offset+metadata+timestamp information.

The second group of Offsetmetadataanderror, from the name of the feeling is Offset+metadata+errorcode style. A look at the code is sure. The constructor of the Offsetmetadataanderror class is to receive these three parameters. Of course, there are 2 auxiliary constructors that just use a lot of default values, and it's no big deal. The Astuple method is to package the three fields into a tuple to return. ToString is no more than said.

After defining the class, Offsetmetadataanderror object has the effect of the factory class, instantiating 4 instances of the class, respectively

Nooffset: Invalid displacement, no meta data (metadata), no error exception

Offsetsloading: Invalid displacement, no metadata, exception in displacement loading

Notoffsetmanagerforgroup: Invalid displacement, no Metadata,consumer no corresponding coordinator anomaly

Unknowntopicorpartition: Invalid displacement, no metadata, unknown topic or partition ID not in valid range [0, numPartitions-1]

Seven, Topic.scala

When I saw Topic.scala, I got a little excited. Finally, we see the code that's a little bit associated with the core functionality. Originally expected topic Code will be very long, unexpectedly and not too much, just simply define an object, a few fields plus a method is done. The name topic, in fact, is more like a validation class for Kafka topic. Let's look at the details below. The first topic legal characters are still letters, numbers, periods, underscores and dashes, and the topic maximum length cannot exceed 255 characters. The only way validate is to verify the legitimacy of a given topic name. In addition, this object also maintains a set save topic, which will be used when the consumer whitelist and blacklist filter are followed. However, it seems that this collection has only one topic:__consumer_offsets, and the entire Kafka code does not have code to insert/delete it, so it is only used internally, the normal topic should not be added to this set.

Eight, Topicandpartition.scala

Following the previous naming style, this class should save a topic and a partition information at the same time, provide various ways to initialize the class instance, and also provide the Astuple method to return in tuples (topic, partition)

"Original" KAKFA common package source code analysis