Understanding the message buffers inside storm

from:http://www.michael-noll.com/blog/2013/06/21/understanding-storm-internal-message-buffers/
When performing performance optimizations on Storm topology, it is helpful to understand how to configure and apply Message Queuing inside storm, and in this article I will explain and demonstrate the worker and internal thread when storm 0.8/0.9 implements communication between workers How it works when executor.
Storm work inter-process message
Terminology: I will use Message/tuple, which represent the same meaning.
When I say internal messaging, it refers to the internal communication of the storm worker process, which is also the traffic that occurs within the same storm Machine/node. This type of communication relies on several message Q based on Lmax disruptor, Lmax is a high-performance inter-thread message communication library. Note that the inter-thread communication within this worker differs from Inter-worker communication, and inter-work communication often occurs between network-based machines. Storm uses ZeroMQ for inter-work communication by default, with experimental Netty-based inter-work communication in Storm 0.9, so that zeromq/ Netty is primarily for tasks within the work that want to send data to a task within the work of another machine in the cluster.
Refer to the following:
Intra-worker Communication/Inter-thread:lmax disruptor on the same node
Inter-worker Communication (Inter-node communication within the network): ZeroMQ or Netty
Inter-topology communication: Storm does not provide the mechanism, you have to find solutions, such as KAFKA/RABBITMQ message system, database, etc.
If you are not sure about the storm worker process, thread executor, and task differences, please refer to understanding the Parallelism of a storm topology.

Graphic
We start with a picture of the details of the truth

The figure above is the internal message queue profile of the storm worker, the work process-related q is marked with red, the worker executor thread-related q is green labeled, although a node has multiple worker processes, For readability only one worker is shown, and the same worker has multiple thread, only one executor thead is shown here.

Detailed description
Now that you have a brief understanding of the message structure in Intra-worker, let's explain it in detail.
Worker process:
To manage incoming/outgoing messages, each worker has a separate receive thread that listens on a TCP port number, which is configured by Supervisor.slots.ports Topology.receiver.buffer.size determines the buffer size that is sent to the executor thread message queue, and again, each worker has a send thread responsible for the transfer from the worker Queue reads messages and sends them over the network to downstream consumer, the size of the transfer queue is configured by Topology.transfer.buffer.size.
Topology.receiver.buffer.size is the maximum number of messages that receive thread can send to executor incomming queue at a time, and these messages are read over the network. This parameter setting is too large may cause a lot of problems, such as heartbeat thread hunger, throughput dips, the default value is 8 elements, and must be a power of 2, this requirement is lmax disruptor indirect decision.
Example:configuring via Java API
Config conf = new config ();
Conf.put (Config.topology_receiver_buffer_size, 16); Default is 8
Topology.receiver.buffer.size is used to compare the size of other buffer sizes in this article and is not used to configure the Lmax disruptor queue size, which is used only to set buffer incomming The size of the message ArrayList, in which case ArrayList is not shared by other thread, it is a local variable of the receive thread. But its content will be used to fill executor incoming Q based on disruptor Q, the size of this disruptor Q must be power of 2, For more detailed information, refer to the Launch-receive-thread in Backtype.storm.messaging.loader
Each element in the Topology.transfer.buffer.size configured transfer queue is a tuple list, and multiple executor send thread will be outgoing from executor Q out and batch outgoing tuples, the default value of this parameter is 1024 elements.
Example:configuring via Java API
Conf.put (Config.topology_transfer_buffer_size, 32); Default is 1024

Executor:
One or more executor thread is controlled within each worker. Each executor thread has his own incoming Q and outgoing Q, as described above, each worker has a dedicated receive thread responsible for moving messages from worker incomming Q to Executor Thread Incomming Q, similarly, each executor has a dedicated send thread responsible for moving messages from executor outgoing q to the transfer q,executor incomming/of the worker. Outgoing q size is configured via Topology.executor.receive.buffer.size/topology.executor.send.buffer.size.
Every executor thread has a separate thread that handles the user's logical Spout/bolt, and there is a send thread from executor outgoing Q in the move message to worker Thransfer Q.
Topology.executor.receive.buffer.size configured the size of executor incomming Q, each element in Q is a tuple list. A tuple is appended to Q in batch, which defaults to 1024 elements, which must be a power of 2, and this requirement is from Lmax disruptor.
Example:configuring via Java API
Conf.put (Config.topology_executor_receive_buffer_size, 16384); batched; Default is 1024
Topology.executor.send.buffer.size configured executor outgoing q size, q Each element is a tuple, this parameter is elements by default, the parameter value must be a power of 2, this requirement from Lmax Disruptor.
Example:configuring via Java API
Conf.put (Config.topology_executor_send_buffer_size, 16384); Individual tuples; Default is 1024

Additional information to refer to:
How to configure storm's internal message buffers
above mentioned parameter default value defined in Conf/defaults.yaml. You can configure the global configuration in the Storm cluster conf/ Storm.yarn overrides the default value, of course you can also configure a topology through the Storm Java API Backtype.storm.Config.
How to configure Storm's parallelism
The correct configuration of the storm's message buffer is closely related to the topology workload, as is the case with topo parallelism configuration, Refer to understanding the Parallelism of a Storm topology configuration prallelism.
Understanding the state of storm topology
The Storm UI is a good entry for monitoring topo metrics, such as showing the capacity of Spout/bolt, These different metrics will help you decide whether to adjust the buffer configuration parameters involved in this article for positive or negative effects on topo performance, refer running a multi-node Storm Cluster for more details.
You can also generate your own app metric and track it with tools such as graphite. Refer to the sending Metrics from Storm to Graphite and installing and Running Graphite via RPM and Supervisord for more information. It might be worthwhile to check out Ooyala's metrics_storm from GitHub, but I haven't done so yet.

Performance Tuning Recommendations
Watch Nathan Marz's vedio Tuning and productionization of Storm
Try starting with the following configuration and observing if it lifted your topology performance C Onf.put (Config.topology_receiver_buffer_size, 8);
Conf.put (Config.topology_transfer_buffer_size, 32);
Conf.put (config.topology_executor_receive_buffer_size, 16384);
Conf.put (config.topology_executor_send_buffer_size, 16384);