Kafka Utils Package Last Post ~ ~ ~
15, Shutdownablethread.scalathe thread abstraction class that can be closed! Inheriting from thread also receives a Boolean variable isinterruptible indicates whether interrupts are allowed. Since it is closed, it must not be a daemon thread, but rather a user thread (which does not block the JVM from shutting down). The methods provided are:1. DoWork:abstract method. Subclasses must implement this method, which should be the name of the specified thread to complete the operation. 2. Initiateshutdown:initiates a shutdown request. First, the CAS is used to determine whether the thread is running, and if so, the isrunning variable that indicates the running state is set to false, based on the interruptible nature of the thread (byisinterruptible) calls the Thread.interupt method to break the thread and return true. 3. Awaitshutdown: Used to initiate a close request (Initiateshutdown) After the method is called to wait for the close operation to complete. The implementation is to call Countdownlatch's await method to wait for the Shutdownlacth object to become 0--shutdownlatch is a closed valve maintained by this class. The program will reduce the object to 0 after the running thread finishes. 4. Run method: A copy of the thread's Run method. This class maintains a isrunning similar to a status flag that controls when a thread exits a loop that performs an operation. Detects the isrunning value before each cycle, and if it is false, exits execution and shutdownlatch the valve to the off state; if true, calls DoWork to perform the actual operation.
16, Throttle.scalaThe main purpose is to limit the execution speed of some operations, in fact, mainly used to clean up the log when the IO speed limit. This class receives a given expected rate (in **/per second, where the * * is not really important, it can be a byte or a number, mainly a throttling rate). The constructor parameters are as follows:
1. Desiredratepersec: Expected rate 2. Checkintervalms: Check interval, Unit MS3. Throttledown: Whether you need to adjust the speed down, that is, reduce the rate of 4. Metricname: Name of the item to be adjusted 5. Units: The unit to be adjusted, the default is byte 6. Time: The class also implements Kafkametricsgroup trait, which you can think of as building a metric object (for example, through Newmeter). The throttle class has only one method: Maybethrottle. This method code writes a lot, one sentence analysis is too boring, I give an example to say: Suppose we want to limit the IO rate, the unit is bytes per second, every 100 milliseconds to check. We want to limit the rate to 10 bytes per millisecond. Now that we have detected a total of 6000 bytes in the 500ms, then the actual rate is 6000/500 = 12 bytes per millisecond, higher than the expected rate, so we want to limit the IO rate, what to do now? Very simple, if it is according to the desired rate, should spend 6000/10 = 600ms, more than the actual cost of 100ms, so the program sleep 100ms to spend more time wasted on the period of the effect of limiting the rate. In a nutshell, this is how the program is implemented:)
17, Time.scalaencapsulates some time-related constants and provides a number of abstract methods for subclasses to implement, such as milliseconds, nanoseconds, and sleep. Finally, it really provides a subclass that implements time trait and implements the previous three methods.
18, Toolsutils.scalaUse the Jopt-simple tool Library to verify that the Host/port format provided by the command-line arguments is correct, Validateportordie method docking the received host:port string to split, filtering out those that do not conform to the host:The entry in port format gets a array[string], if the number of the array does not match the split array length, it indicates that there are unqualified items,Directly prints the calling method and ends the program.
okay! We're getting close to winning! But the thought of the KAFKA.UITLS package also has 3 extra-long source code files on the head pain. Let's have a bite.
19, Utils.scalaAs the note in this file suggests, the Help functions in this file are the most commonly used functions and are not limited to Kafka logic services. So if you want to add a function to this file, make sure that the function is well documented and that it must not be used only in a particular area, but in the most common scenarios. Okay, anyway, is a separate function, or the usual, one said:
1. Runnable:a function that receives a non-return value is encapsulated into a new Java Runnable object that is returned. 2. Daemonthread: Create a daemon thread for background execution. Note that just create, do not open the thread. There are 3 ways to do the same.3. Newthread:creates a new thread, which determines the name of the thread, the action performed, and whether it is a daemon thread4. Readbytes:reads a given bytebuffer and a given displacement to a byte array5. Loadprops:loads a. properties file under a given path into a properties object6. Openchannel:open a file channel according to the variability, if you can modify it, open it in the Randomaccessfile method, if it is not modifiable, open it in the FileInputStream method .7. Swallow:performs a given operation (passed as a function parameter), records any exception but never throws an exception, but swallows the exception8. Equal:compare two segments bytebuffer--first compare position, then remaining, and finally each byte inside9. ReadString:read the contents of the Bytebuffer to generate a stringTen. Croak:printing error messages and shutting down the JVMOne . RM:to delete a file or subdirectory recursivelyRegistermbean.:registers the given Mbean with the platform Mbean server. If you have already registered, you will first uninstall the Mbean and then reregister. The method does not throw any exceptions, just simple drops return false to indicate registration failedUnregistermbean.:cancels the registration of the Mbean. If you don't register, you don't do anything .Readunsignedint.:there are altogether 2 methods, one is to read the 4 byte integer from the current position of the Bytebuffer and update the displacement, and the other is to read the 4 byte integer from the given position without modifying the position informationWriteunsignedint.:2 methods are also available.:one is to write a long type as an integer of 4 bytes, regardless of the problem of overflow, and the second method is to do the first method at a given position BytebufferCRC32.:computes the CRC32 checksum of a byte arrayhashcode.:calculates the hashcode of a given parameter itemGroupBy.:The key that is computed by the given function is grouped, the value of the same key is put into a list[v], and the whole is returned as a hashmap. But it seems that this function has not been used:(. Read:reads the buffer of the given readablebytechannel to a bytebuffer, or returns the number of bytes read if the read fails to throw a Eofexception exceptionNotnull.:a method with a generic type that returns the value directly if the given value is a null throw exceptionStackTrace.:Gets the stack trace information for the given exception class, returned as a stringParsecsvmap.:receives a comma-delimited pair of key/value, similar to Key1:Value1,key2:value2,... Resolves to a hashmap using a regular expression and returns. The method also has a form that is returned as a sequence of strings, that is, an array of stringsCreateObject.: Creates an instance 24 for the class represented by a given class name through the reflection mechanism. Nullorempty: Determines whether the given string is empty. However, the implementation has a problem, if it is more than one space, the method will return FALSE25. Loopiterator: is actually an infinite traversal iterator that generates an infinite stream stream for the incoming collection and returns its iterator 26. Readfileasstring: Reads the contents of the file and returns 27 as a string. ABS: Calculates the absolute value of an integer. If it is the smallest integer returned 0--differs from Java implementation by 28. Replacesuffix: Replace the suffix of the string if it cannot find the suffix to be replaced directly throws exception 29. CreateFile: Creates a file 30 based on the given path. Asstring: Converts a Properties object to a string to return 31. Readprops: Reads the property from the given string and returns a PROPERTIES32. ReadInt: Reads a integer33 from the specified offset in the byte array. Inlock: Performs a function 34 in the case of a lock. Inreadlock/inwriterlock: Acquires a read and write lock with the given lock, and then executes the function body 35. Jsonescapestring: Escapes certain characters in a string, such as \b to \\b36. Duplicates: Returns duplicate items in a list
20, Verifiableproperties.scalaThis class encapsulates the properties object while maintaining a hashset representation of the property name collection. The specific methods are: 1. ContainsKey: Determines whether a key2 is included. GetProperty: Adds property names to the property name collection and then returns 3 after getting a property value from props. GetString: First detects if this property is included, and if it is included, call GetProperty to return the property value 4. GetInt: Gets the property value 5 for an integer type. Getintinrange: Returns a property value of type Integer, but the value must be within a given range of 6. Getshort: Returns the property value 7 for a short type. Getshortinrange: Returns a short type of property value, but the value must be within a given range of 8. Getlong: Returns a long property value of 9. Getlonginrange: Returns a long property value, but the value must be within a given range of 10. GetDouble: Returns the property value 11 of a double type. Getboolean: Returns a Boolean property value of 12. Getmap: Resolves a map[string, String] from a list of properties and returns 13. Getcompressioncodec: Reads the codec information from the property list. The method also supports parsing the ordinal and name of the codec, and returns the corresponding CODEC14. Verify: The main thing is to verify that each property in the Properties object is in the property name collection, even if it's not the same as printing a log.
21, Zkutils.scalaFinally to the last file in the bag. This file code is very long and defines three class and two object. Let's start with the simple: 1. Zkconfig class: This class defines the zookeeper configuration information, and its constructors receive a Verifiableproperties object. Zookeeper configuration information maintained by Kafka includes: a CSV-formatted zookeeper connection string, the most general session timeout, the maximum connection timeout, and a zookeeper follower is allowed to lag behind leader the maximum time interval, which defaults to 2 seconds by 2. Zkgroupdirs class: The group in the name refers to the Kafka consumer group, because consumer in Kafka belongs to a consumer group. In this class, 3 methods are defined to return the root path of the consumer, the path of the consumer group, and the path to the ID of the consumer group 3. Zkgrouptopicdirs class: It inherits the Zkgroupdirs class and provides 2 new methods for calculating the displacement and owner information of the consumer group. As shown in: Zkutils.scala also defines a Zkstringserializer object that is used to serialize/deserialize a saved byte array in the zookeeper. The two methods provided are also intuitive: The Serialize method converts a string into a byte array, and the Deserialize method converts a byte array into a string.Well, the last object did not say. Zkutils ObjectMany constants are defined first, which are the zookeeper paths used by different components of the Kafka, such as the root path of consumer,/consumers, etc. It is worth noting that currently Kafka does not support our custom values for these paths. This object defines a lot of methods, although it is very intuitive, but it is worth to simply browse through: 1. Gettopicpath: Gets the ZK path of topic 2. Gettopicpartitionspath: Gets the ZK path of the topic Partition 3. Gettopicconfigpath: Gets the ZK path of topic CONFIG. This path was introduced in 0.8.1, and the ZK information under this path is updated when the topic partition assignment is created or modified 4. Getdeletetopicpath: Gets the ZK path of the deleted topic 5. Readdatamaybenull: Returns the (Controller name, state) tuple under the given/controller path, if the ZK node cannot be found directly back (None, null stat) 6. Getcontroller: Call the Readdatamaybenull method to get the controller name, or return an integer type controller ID if null throws an exception directly ( By calling the Kafkacontroller.parsecontrollerid method) 7. Gettopicpartitionpath: Back/brokers/topics/<my-topic>/partitions/<my-partition-id> Such a form of ZK path8. Gettopicpartitionleaderandisrpath: Back/brokers/topics/<my-topic>/partitions/<my-partition-id>/state such a form of ZK node 9. GetChildren: Returns all sub-path names under the given path path 10. Getsortedbrokerlist: For/borkers/ids All brokers are sorted by broker ID and a list is returned 11. Getchildrenparentmaynotexist: Gets all the subdirectory names for the given path, if the node represented by path does not exist return nil 12. Getbrokerinfo: Reads the data/brokers/ids/< the given broker id> node and, if present, calls Broker.createbroker to create a broker instance to return, otherwise returns NONE13. Getallbrokerincluster: Gets the broker list under/brokers/ids and sorts, iterates through the list to create a broker object for each broker that is not none, and finally encapsulates the return 14 in a seq. Getleaderandisrforpartition: Leader node and all ISR and epoch information are queried for the specified topic and its partition, Then extract the leader and ISR information encapsulated as a Leaderandisr object return--LEADERANDISR is a case class that encapsulates the leader information and its ISR list information 15. Makesurepersistentpathexists: Make sure that the given path exists in zookeeper if it does not exist to create the path 16. Setupcommandpaths: Call makesurepersistentpathexists to ensure that all paths defined at the beginning of the class already exist in 17. Getleaderforpartition: Gets the leader ID of one of the partition for a topic. For example, the method should return 018. Getepochforpartition: Reads the epoch information under a section of a topic. If you do, you should return the value of Leader_epoch, which is 2019. Getinsyncreplicasforpartition: Reads all ISR lists under a topic partition, and if so, should returnThe value of the back ISR, also 0--author's Kafka is a single-node cluster, so can not see too big difference 20. Getreplicasforpartition: Reads a set of copies under a topic partition, if yes topic name is Log-topic, the partition number is 0, should return [0]--if the production environment should be a set of assigned replicas 21. REGISTERBROKERINZK: Creates a temporary node 22 on zookeeper for the given broker ID. Getconsumerpartitionownerpath: Returns the ZK path 23, such as/consumers/console-consumer-33497/owners/log-topic/partitionid. Leaderandisrzkdata: Encodes the given Leaderandisr object and epoch into a JSON-formatted string of 24. Replicaassignmentzkdata: Encodes a set of replica into a JSON-formatted string: "Version": 1, "Partitions": {...} Createparentpath: Creates a persistent node 26 represented by the parent path. Createephemeralpath: Creates a temporary node 27 based on the given data and path. Createephemeralpathexpectconflict: As the name implies, create a temporary node if there is already a throw exception 28. Createephemeralpathexpectconflicthandlezkbug: A more rigorous way to create a ZK temporary node--primarily to handle the bug29 of zookeeper session timeouts. Createpersistentpath: Cascade creates a persisted node 30. createsequentialpersistentpath: Creates a sequential persistent node (persistent sequential) 31. updatepersistentpath: Updates the value of a persistent node. If necessary, create its parent path, but not the exception that the node already exists 32. conditionalupdatepersistentpath: if the given path and version have no problem updating the node data, return (true, new version), Otherwise returns (false,-1). In addition, if you catch a missing connection exception (for example: Connectionlossexception), Zkclient will automatically retry, but it is possible that the last update operation was successful, so the updated version number no longer matches the expected version number, causing the method to fail, so the method needs to handle this situation, The practice is to provide a optional function to do further inspection 33. conditionalupdatepersistentpathifexists: It is also to determine the condition before deciding whether to update the operation. Returns (true, new version) if successful, otherwise returns (false,-1). Also if path does not exist directly throws an exception end 34. updateephemeralpath: Updates the value of a persisted node, creates its parent path if necessary, but does not throw nodeexistsexception35. Deletepath: Deletes the given path node, returns true to indicate success, or returns false36. deletepathrecursive if the node does not exist: the given path37 is removed recursively. Maybedeletepath: Creates a ZK client connection based on the given zookeeper client connection string, sets the session timeout and the connection timeout is 30 seconds, then recursively deletes the given path, then closes the Zkclient object. If there are any exceptions do not do any processing, directly swallow 38. readdata: Read the data of the path node, return (data, status information) tuple 39. Pathexists: Checks if the given path exists 40. Getcluster: Iterates through all the brokers under/brokers/ids, creates a broker object and joins to an cluster object instance (cluster holds all currently available brokers), and then returns CLUSTER41. getpartitionleaderandisrfortopics: Given a set of Topicandpartition objects (Topicandpartition is a topic plus a partition information), For each query out corresponds to this topic partition of the Leaderisrandcontrollerepoch and encapsulated in a hashmap return-simply to get a topic a partition leader information, ISR information, Leaderepoch Information and Controllerepoch information 42. getreplicaassignmeNtfortopics: Gets the copy of all partitions for a set of topic broker number 43. getpartitionassignmentfortopics: Returns the hashmap[topic name, map[ The topic under each area code, a set of copies]] in this form map44. getpartitionsfortopics: Returns the Map[topic name, the topic next group partition] Such a form of map45. getpartitionsbeingreassigned: Read all new replicas for all topic and their partitions in list 46. Parsepartitionreassignmentdatawithoutdedup: Creates a tuple sequence based on a given JSON string, each of which is in (Topicandpartition, Seq[int]) format, Specifies the replica number that corresponds to a partition of a topic. In fact, given a JSON string, which specifies which replica number belongs to which topic which partition, and then extracted to do a seq save. Another variant of the method is to return a map, in short, the data structure is different. 47. parsetopicsdata: Extracts topic information from a given JSON string into a list to return 48. Getpartitionreassignmentzkdata: Generated {"Version": 1, "Partitions": {"topic", "Partition", "replicas": { * *, *}}} This form of JSON string 49. updatepartitionreassignmentdata: Call the Getpartitionreassignmentzkdata method to generate a JSON string, Then call the Updatepersistentpath method to update zookeeper50. getpartitionsundergoingpreferredreplicaelection: Read/admin/ Preferred_replica_election All information, returns a set of topicandpartition51. deletepartition: delete/bRokers/ids/brokerid node, and remove/borkers/topics/brokerid node 52. getconsumersingroup: Get/consumers/group/ All node names under IDs 53. getconsumerspertopic: Gets all consumser54. getbrokerinfo for each topic: Gets broker information 55 based on broker ID. Getalltopics: Get all the topic encapsulated into a SEQ 56. getallpartitions: Get all partition of all topic, flatten into a set, Each of them is a Topicandpartition object
Original KAKFA utils source code Analysis (iii)
