Storm topology an optimization idea

Original: https://my.oschina.net/jerrysearch/blog/201373

A discussion on distributed environment a way to optimize storm topology is certainly not all that is included in the optimization, but it must be part of it.

First assume an environment, three machines, CPU core 4,net miles net

Topology

Component	Name	Executor Latency (ms)
Spout	S	-
Bolt	A	0.5
Bolt	B	0.1
Bolt	C	0.5
Bolt	D	1.0
Bolt	E	0.2

Executor represents the time the data goes into the bolt Execute method execution, storm is sampled by sampling, the total number of samples will certainly have errors, but the average we think for a while, this is equivalent to a random selection of 1W 1, how to choose is 1, So for the moment we assume that executor latency credible, and finally we validate our assumptions in turn by actual efficiency.

Assuming that we submitted a topology with a parameter, after running the UI to view parameters such as tables, we observed that a,c two bolt delay at 0.5ms,b bolt delay in 0.1ms,d bolt delay in 1.0ms,e Bolt delay at 0.2ms, so can be considered performance: b=2e=5a=5c=10d, so we improve the efficiency of the entire topology can adjust the degree of parallelism of each bolt, we set the degree of parallelism of B is X, can be derived X+2x+5x*2+10x=all executor , if there is a blockage in a bolt, we can increase some degree of parallelism, the environment of three machines, the maximum degree of parallelism is 3*4=12,x "= 1, so x=1, thus can be obtained, B Bolt parallelism is 1, so the processing efficiency of about =1000/0.1*1=1w per second.

In addition, in storm each worker has a thread pool, the pool size defaults to the core number, we can increase the pool size according to the situation, improve CPU utilization.

Other parameters are not mentioned here, according to the actual situation can be adjusted as appropriate.

Summarize:

As you can see, the executor latency in the UI is basically trustworthy, and we can refer to this parameter when tuning.