Storm Internal Message Cache

This paper is translated from http://www.michael-noll.com/blog/2013/06/21/understanding-storm-internal-message-buffers/

It is helpful to understand the configuration of storm internal Message Queuing when Storm tuning is in progress. This document will describe the message communication inside a worker within the Storm 0.8/0.9 version.

Storm worker process internal message transfer

The word "internal message" here refers to a message inside a worker process on a single node. This communication relies on a variety of LMAX disrupter (high-performance multithreaded message store) support for Message Queuing within storm.

Multi-threaded communication within a single worker process differs from communication between multiple worker processes. The latter usually spans networks or machines, and Storm uses Zero Mq/netty to support it.

• Intra-worker Communication in Storm (Inter-thread on the same storm node): LMAX disruptor
• Inter-worker communication (Node-to-node across the network): ZeroMQ or Netty
• Inter-topology communication:nothing built into Storm, you must take care of this yourself with e.g. a messaging system s Uch as KAFKA/RABBITMQ, a database, etc.

Case description

Summary of storm internal message queues. The worker process's message queue is identified in red, and the worker's internal Executer message queue is identified as green. In order to be readable, there is only one worker and one executor, and in reality there are often more than one worker, and there may be multiple executors in a worker.

Describe worker processes in detail

To manage incoming and outgoing messages, each worker process has a receive thread that listens on the TCP port (configured by Supervisor.slots.ports). Similarly, each worker has a send thread responsible for sending messages read from the transfer queue to downstream consumers.

• The is the maximum number of messages that be topology.receiver.buffer.size batched together at once for appending to an executor ' s incoming Queu E by the worker receive thread (which reads the messages from the network) Setting This parameter too high may cause a lot of problems ("heartbeat thread gets starved, throughput plummets"). The default value is 8 elements, and the value must being a power of 2 (this requirement comes indirectly from LMAX disruptor ).

• // example:configuring via Java API New  //  default is 8

• Each element of the transfer queue configured with is topology.transfer.buffer.size actually a list of tuples. The various executor send threads would batch outgoing tuples off their outgoing queues onto the transfer queue. The default value is 1024x768 elements.

• // example:configuring via Java API // default is 1024x768

Executors

A worker process controls one or more executer threads. Each executor has its own incoming queue and outgoing queue

Each executor has a thread responsible for logic processing Spout/bolt, one receiving message thread, and one sending message thread.

• The is the size of the topology.executor.receive.buffer.size incoming queue for an executor. Each element of the this queue is a list of tuples. Here, tuples is appended in batch. The default value is 1024x768 elements, and the value must be a power of 2 (this requirement comes from LMAX disruptor).

• // example:configuring via Java API // batched; default is 1024x768

• The is the size of the topology.executor.send.buffer.size outgoing queue for an executor. Each element of the this queue would contain a single tuple. The default value is 1024x768 elements, and the value must be a power of 2 (this requirement comes from LMAX disruptor).

• // example:configuring via Java API // individual tuples; default is 1024x768

Storm Internal Message Cache