The problem of velocity suppression in storm spout

Forwarding please specify the original address: http://www.cnblogs.com/dongxiao-yang/p/6031398.html

Recently assisted colleagues to optimize a concurrent consumption Kafka data used to calculate the task, the pressure measurement process found that there are two spout corresponding topic consumption rate is significantly lower than the other topic indicators, each spout allocated 10 concurrent consumption speed to about 1w completely on the go, By monitoring the buried point to analyze the spout and downstream of the bolt code block inside the code execution time is not higher than the rest of the normal consumption of topic corresponding spout components.

Finally can only pick out the problematic code to do a demo to test, found that the Nexttuple collector. Emit () This method of call logoff, only to retain the logic of reading Kafka after the demo program Consumption Kafka speed is also stuck at a very low speed, to see the problem program code nexttuple call logic is probably as follows

if (booleanfunction)

{

Collector.emit (...)

}

Where booleanfunction refers to a method that executes a business code and returns a Boolean value, presumably this method does not return true on the actual line each time to enter the link that calls the emit method,

Modify the code as follows

if (booleanfunction)

{

Collector.emit (...)

}

Else

{

Collector.emit (...)

}

The equivalent of every nexttuple call will run the emit method, the task back online after 10 spout consumption easy to break through 30w+.

The cause of the problem is that the emit method updates a Emitted-count variable value in memory after each execution of the Nexttuple code as the storm's spout executes. If the spout discovery Emitted-count is consistent with the value after the last call, the Nexttuple function does not send a message, and the Emitempty method of Spout-wait-strategy is called. By default this method will sleep for a millisecond. so in the absence of emit, nexttuple theoretically the biggest call frequency is 1000/s

。

Resources

1 storm spout emit problem

2 Storm Source Analysis Chapter 10th 10.3.5 message loop

The problem of velocity suppression in storm spout