Storm-kafka Source Code parsing

Storm-kafka Source code parsing

Description: All of the code in this article is based on the Storm 0.10 release, which is described in this article only for kafkaspout and Kafkabolt related, not including Trident features. 

Kafka Spout

The Kafkaspout constructor is as follows:

Public Kafkaspout (Spoutconfig spoutconf) {
    _spoutconfig = spoutconf;
}

Its construction parameters come from the Spoutconfig object, and all parameters used in spout are derived from the object. The object parameters are described as follows:

  spoutconfig

Spoutconfig inherits from Kafkaconfig. All parameters and descriptions within the two classes are as follows:

/ **
 * Kafka address and partition correspondence information
 * If Kafka's partition information and address information are clear, you can use StaticHosts directly
 * But this object parameter is difficult to construct and requires a lot of information, so we don't use it in general.
 * We mainly use instances of ZKHosts. Can set Zookeeper address and other information in it, and then get kafka metadata
 * For the parameter information of ZKHost, see the following paragraph.
 * Required parameters
 ** /
public final BrokerHosts hosts;
/ **
 * Topic queue name to read from kafka
 * Required parameters
 ** /
public final String topic;
/ **
 * Kafka client id parameter, this parameter generally does not need to be set
 * The default value is kafka.api.OffsetRequest.DefaultClientId ()
 * Empty string
 ** /
public final String clientId;
/ **
 * The amount of data obtained by each Kafka Consumer request
 * Data will not be acquired until the data obtained is consumed
 * Default 1MB
 ** /
public int fetchSizeBytes = 1024 * 1024;
/ **
 * Kafka SimpleConsumer client and server connection timeout
 * Unit: millisecond
 ** /
public int socketTimeoutMs = 10000;
/ **
 * Consumer timeout
 * Unit: millisecond
 ** /
public int fetchMaxWait = 10000;
/ **
 * The size of the socket buffet that the Consumer uses to get data from the network IO,
 * Default 1MB
 ** /
public int bufferSizeBytes = 1024 * 1024;
/ **
 * This parameter has two functions:
 * 1: declare output data fields declareoutputFileds
 * 2: Deserialize the data read from kafka, that is, convert the byte array into a tuple object.
 * Both the key and message that kafka stores data are more concerned about, you can use KeyValueSchemeAsMultiScheme,
 * If you don't care, you can use SchemeAsMultiScheme
 * The default interface implementation generally only outputs one field or two fields. In many cases, we need to read the data directly from kafka, then parse each field, and then simply process and then emit
 * At this time, it is recommended to implement the MultiScheme interface yourself
 * Required parameters
 ** /
public MultiScheme scheme = new RawMultiScheme ();
/ **
 * After the topology is submitted, KafkaSpout will read the previous offset value from zookeeper in order to continue reading data along the last position.
 * KafkaSpout will check if the topology ID and the topology id stored in zookeeper are the same.
 * If different, and ignoreZkOffsets = true, then the data will be read from the startOffsetTime parameter position
 * Otherwise, continue reading data along the offset position saved in zookeeper.
 * In other words, when ignoreZkOffsets = true, kafkaspout can only guarantee that the topology will not be killed. When the worker process exits abnormally, it will continue to read data along the last read position. When the topology is resubmitted At that time, data will be read from the earliest position in the queue.
 * In this case, there will be a problem of repeatedly reading data, so in a formal scenario, this parameter should still be set to false. To ensure that any scene data is read only once.
 ** /
public boolean ignoreZkOffsets = false;
/ **
 * When the topology is submitted for the first time, and the corresponding offset is not stored in zookeeper, the offset position read from kafka by default. By default, data is read from the earliest position in the queue, that is, data is read from the earliest position in the queue.
 ** /
public long startOffsetTime = kafka.api.OffsetRequest.EarliestTime ();
/ **
 *
 * If the current (offset value-the minimum value of failed offsets) <maxOffsetBehind
 * Then all offset values greater than maxOffsetBehind in the failed list will be cleared.
 * This is to prevent too many failings and too many retransmissions to cause memory overflow
 * However, to ensure that data is not lost by default, the maximum value set by maxOffsetBehind
 ** /
public long maxOffsetBehind = Long.MAX_VALUE;
/ **
 * After KafkaSpout is initialized, the offset of the last record read from zookeeper is used
 * Failed to get data from kafka, after returning offsetOutofRange error,
 * Whether to use startOffset to retrieve data from the earliest position in the queue.
 * OffsetOutofrange generally occurs in scenarios where topics are reconstructed and fragments are deleted.
 ** /
public boolean useStartOffsetTimeIfOffsetOutOfRange = true;
/ **
 * Metric monitoring information collection interval
 ** /
public int metricsTimeBucketSizeInSecs = 60;
/ **
 * KafkaSpout saves the address of the zookeeper
 * This attribute is independent to prevent the offset save location from being in the kafka cluster
 * If kafka and storm are in a cluster, this attribute can be ignored
 ** /
public List <String> zkServers = null;
/ **
 * KafkaSpout zookeeper port to save offset
 * If kafka and storm are in a cluster, this attribute can be ignored
 ** /
public Integer zkPort = null;
/ **
 * offset path saved in zookeeper
 * Path calculation method is: $ {zkRoot} / $ {id} / $ {partitionId}
 * Required parameters
 ** /
public String zkRoot = null;
/ **
 * KafkaSpout stores different client distinguishing flags for offset
 * It is recommended to use fixed and different parameters for each topology to ensure that data can be read from the last location after the topology is resubmitted
 * If two topologies share the same id, they may be read repeatedly
 * If a dynamically generated uuid is used in the topology as the id, then each time the topology is submitted, data will be read from the beginning of the queue
 * Required parameters
 ** /
public String id = null;
/ **
 * Offset refresh interval to zookeeper
 * Unit: millisecond
 ** /
public long stateUpdateIntervalMs = 2000;
/ **
 * Retry policy related parameters after data transmission fails
 ** /
public long retryInitialDelayMs = 0;
/ **
 * Retry policy related parameters after data transmission fails
 ** /
public double retryDelayMultiplier = 1.0;
/ **
 * Retry policy related parameters after data transmission fails
 ** /
public long retryDelayMaxMs = 60 * 1000;

Information such as the zookeeper address where the Kafka cluster resides is saved in Zkhost 

zkhost

/**
 * Kafka cluster zookeeper address, allow to include chroot
 * For example: 192.168.0.10:2181,192.168.0.11:2181,192.168.0.12:2181/kafka
 **/public
String brokerzkstr = null;
/**
 * Kafka the broker metadata address in the cluster
 * default is/brokers
 * If Chroot is configured, then/kafka/brokers
 * This and the KAKFA server configuration default is the same, If the server has a default configuration, this property can also use the default value
 **/public
String brokerzkpath = null;//e.g.,/kafka/brokers
/**
 * Kafka Broker partition information Refresh interval,
 * units: Seconds
 * When the Kafka has a broker node reboot or the partition information changes and the data read fails, the
 partition information refresh will be triggered again
 **/ public
int refreshfreqsecs = 60;

Kafkaspout Initialization

public void open (Map conf, final TopologyContext context, final SpoutOutputCollector collector) {
        _collector = collector;

        Map stateConf = new HashMap (conf);
        / *
         * offset zookeeper address of the save location
         * If the address is empty, the zookeeper of the Storm cluster is used by default
         * /
        List <String> zkServers = _spoutConfig.zkServers;
        if (zkServers == null) {
            zkServers = (List <String>) conf.get (Config.STORM_ZOOKEEPER_SERVERS);
        }
        Integer zkPort = _spoutConfig.zkPort;
        if (zkPort == null) {
            zkPort = ((Number) conf.get (Config.STORM_ZOOKEEPER_PORT)). intValue ();
        }
        stateConf.put (Config.TRANSACTIONAL_ZOOKEEPER_SERVERS, zkServers);
        stateConf.put (Config.TRANSACTIONAL_ZOOKEEPER_PORT, zkPort);
        stateConf.put (Config.TRANSACTIONAL_ZOOKEEPER_ROOT, _spoutConfig.zkRoot);
        // Save offset information to zookeeper
        _state = new ZkState (stateConf);

        // kafka cluster connector
        _connections = new DynamicPartitionConnections (_spoutConfig, KafkaUtils.makeBrokerReader (conf, _spoutConfig));

        // using TransactionalState like this is a hack
        int totalTasks = context.getComponentTasks (context.getThisComponentId ()). size ();
        if (_spoutConfig.hosts instanceof StaticHosts) {
            _coordinator = new StaticCoordinator (_connections, conf, _spoutConfig, _state, context.getThisTaskIndex (), totalTasks, _uuid);
        } else {
        // Read kafka's broker information from zookeeper, and only save the partition information needed by its own instance
            _coordinator = new ZkCoordinator (_connections, conf, _spoutConfig, _state, context.getThisTaskIndex (), totalTasks, _uuid);
        }

        // Two metrics monitoring information, ignore
        context.registerMetric ("kafkaOffset", new IMetric () {...}, _spoutConfig.metricsTimeBucketSizeInSecs);

        context.registerMetric ("kafkaPartition", new IMetric () {...}, _spoutConfig.metricsTimeBucketSizeInSecs);
    }

The above is the initialization method of Kafkaspout, the main is to complete the management of their own partition information refresh.
Here's the problem of creating 3 zookeeper client connections, one for reading from Kafka, one for saving offset, one for metrics monitoring information, and 3 threads per zookeeper client connection, so There are 9 zookeeper threads in the light of a kafkaspout. When there are multiple spout instances in the worker process, more threads are generated, which consumes performance, and it is recommended that the zookeeper connection be merged.

The system uses the Kafkautils.calculatepartitionsfortask method to obtain the list of partitions that it needs to manage:

for (int i = Taskindex; i < numpartitions; i + = totaltasks) {
            Partition taskpartition = Partitions.get (i);
            Taskpartitions.add (taskpartition);
        }

Where the taskindex corresponds to the ordinal of its own spout instance, such as the spout concurrency of 3, then this spout instance may be 0,1,2. When the Kafka topic has 5 partitions, the first spout instance manages the 0,3 partition, the second spout instance manages the partition numbered 1,4, and the third spout instance manages the partition numbered 2.
TaskID is stored in the context parameter of the open method of spout. Context.getthistaskindex ()

  kafkaspout How to read the data from the Kafka and send

Kafkaspout mainly reads the data and emit in the Nexttuple method.

public void nextTuple () {
        // Get the list of partitions managed by its own instance
        List <PartitionManager> managers = _coordinator.getMyManagedPartitions ();
        for (int i = 0; i <managers.size (); i ++) {

            try {
                // _ currPartitionIndex is always smaller than the size of the manager
                // in case the number of managers decreased
                _currPartitionIndex = _currPartitionIndex% managers.size ();
                // Get data and emit
                EmitState state = managers.get (_currPartitionIndex) .next (_collector);
                / *
                 * Check the data sending status
                 * If no data has been retrieved or the retrieved data has been committed
                 * Then increase the _currPartitionIndex value, and then you can read data from the next partition.
                 * /
                if (state! = EmitState.EMITTED_MORE_LEFT) {
                    _currPartitionIndex = (_currPartitionIndex + 1)% managers.size ();
                }

                / *
                 * If there is still data without an emit, exit the loop and wait for the next nexttuple call
                 * Then still fetch data from the current partition and emit
                 * /
                if (state! = EmitState.NO_EMITTED) {
                    break;
                }
            } catch (FailedFetchException e) {
                LOG.warn ("Fetch failed", e);
                _coordinator.refresh ();
            }
        }
        // Regularly save offset data to zookeeper
        long now = System.currentTimeMillis ();
        if ((now-_lastUpdateMs)> _spoutConfig.stateUpdateIntervalMs) {
            commit ();
        }
    }

Data Send status Emitstate there are altogether three states emitted_more_left
The last data that was taken is not emit finished emitted_end,
The last data fetch is all emit complete no_emitted
No data was taken, no data available for emit

Take a look at the Partitionmanager.next method, which contains how to get the data emit

public EmitState next (SpoutOutputCollector collector) {
        // If the queue waiting for sending is empty, then fetch the data from kafka again
        if (_waitingToEmit.isEmpty ()) {
            fill ();
        }
        while (true) {
        // Get the first data from the queue waiting to be sent
            MessageAndRealOffset toEmit = _waitingToEmit.pollFirst ();
            // If there is no data to send, then the return status is no data that can be emitted
            if (toEmit == null) {
                return EmitState.NO_EMITTED;
            }
            // According to the implementation of KeyValueSchemeAsMultiScheme interface, convert the data obtained from kafka to tuple
            Iterable <List <Object >> tups = KafkaUtils.generateTuples (_spoutConfig, toEmit.msg);
            if (tups! = null) {
            // Send all tuples, because one Kafka data may correspond to multiple storms
                for (List <Object> tup: tups) {
                    collector.emit (tup, new KafkaMessageId (_partition, toEmit.offset));
                }
                break;
            } else {
            // If the tuple conversion fails, return null, tell Storm directly that the bar has been processed successfully, that is, ignore the data error
                ack (toEmit.offset);
            }
        }
        / *
         * Take one piece of data from the waiting queue each time and deserialize and emit,
         * Then determine if there is data in the waiting queue,
         * If there is data, tell spout that the data has not been sent yet, do not switch partitions
         * If the data has been sent, tell spout that the data has been sent and you can switch to the next partition.
         * /
        if (! _waitingToEmit.isEmpty ()) {
            return EmitState.EMITTED_MORE_LEFT;
        } else {
            return EmitState.EMITTED_END;
        }
    }

When there is a failure to send data, the failed data will be re-added to the _waitingtoemit queue, which will create a problem, that is, when the data sent to fail, Kakfaspout will always read only one partition, the day before the partition will not be read, resulting in uneven data consumption problems.

In 0.9.6 old version of the time Yo a problem, is that when more data emit failure, there will be a lot of data in the retry, and then retry the continuous timeout, and constantly rejoin the retry list, resulting in a data transmission of the dead loop. This problem is also the offset timeout problem. See Storm-643, the problem is currently resolved in the latest version. Kafkabolt

Kafkabolt is relatively simple, and version 0.10 uses the old Producer API.
Storm all the configuration properties, are preserved in the kafka.broker.properties, this requires that in the submittopology time, put a kafka.broker.properties attribute in the topologyconf, form a map set map structure. One thing that's bad about this is that the data in a topology can only be written to a Kafka cluster and not supported by colleagues writing to multiple Kafka clusters. However, this has been resolved in the 0.11 new version, and Kafka.broker.properties is used as a local variable to store different configuration properties in different bolt instances.
The data is written in the following ways:

public void execute (Tuple input) {
        if (Tupleutils.istick (input)) {
          collector.ack (input);
          Return Do not try-to-send ticks to Kafka
        }

        K key = null;
        V message = NULL;
        String topic = NULL;
        The key value of the try {
            //message, different values correspond to different distribution methods in Kafka, this is described in the FAQ for Kafkabolt.
            key = Mapper.getkeyfromtuple (input);
            Message body msg
            = mapper.getmessagefromtuple (input);
            Topic name
            TOPIC = topicselector.gettopic (input);
            if (topic! = NULL) {
                producer.send (new keyedmessage<k, v> (topic, key, message));
            } else {
                Log.warn ( "Skipping key =" + key + ", topic selector returned null.");
            Collector.ack (input);
        } catch (Exception ex) {
            collector.reporterror (ex);
            Collector.fail (input);
        }
    }

Storm-kafka FAQ Kafkaspout

The relationship between the number of Kafkaspout Excutor and the number of Kafka topic partitions

When the executor concurrency is greater than the number of topic, there are spout instances that can read the data, and some spout instances cannot read the data.
When the executor concurrency is less than the number of topic, there will be a spout instance corresponding to multiple partitions, Kafka will first take the data from a partition, and when the data obtained is emit, the data will be fetched from the next partition.
When the executor concurrency equals the number of topic, a spout instance corresponds to a partition. In practical applications, we also recommend this configuration method. How to read data from Kafka, how much data to read each time
According to the configuration of the Fetchsizebytes parameter, 1MB data is taken by default. How data read failures are handled
Kafkaspout A retry queue is saved inside each partitionmanager, and when data is sent to fail, join the retry queue and resend until it succeeds.
A memory overflow problem is caused by excessive number of failed by Maxoffsetbehind parameters. Topic does not exist how to handle
Direct error. Topology resubmit, will continue to read data at the last location
When resubmitted, the data will continue to be read at the last location as long as the ID parameter is unchanged. What to do if the offset position of the Kafka saved in the zookeeper is incorrect.
Throws a Offsetoutofrange exception and then defaults to reading the data from the earliest position of the Kafka partition queue. Can read data from multiple topic in one spout.
No, in version 0.10, in version 0.11, it is supported to match the topic name in regular fashion, and the data can be read from all topic that satisfy the regular condition. Topic partition Primary and standby information changes, how to deal with

Throws an exception, then immediately updates the partition information and reads the data again. 

Kafkabolt

Write data, Kafka topic does not exist what to do.

If the KAKFA server allows automatic creation of topic, topic is created automatically.
If auto-creation is not allowed, then an exception is thrown to write the data to the specified partition.
Depends on the interface implementation of the Tupletokafkamapper.
Kafka version 0.10 is using the new producer API for the api,0.11 version of old producer
For Old Producer
If key = = NULL, then in the Kafka, the Random inch of a partition to write the data, then as long as not restart, it will be written to this partition data
If key. = NULL, the partition ID is computed in the Utils.abs (key.hashcode)%numpartitions rule when the data is written
For new Producer
If key = null, then an incrementing int value is used, incremented each time the data is sent, then executes Utils.abs (NextValue)%availablepartitions.size (), and the data write is balanced.
If key. = NULL, the partition is computed according to the rules%numpartitions Utils.abs (UTILS.MURMUR2 (Record.key ())).
Of course, the New Producer API can also manually specify the partition ID.