Storm (v) topological parallelism

Apache Storm distributed cluster main node consists of control node (Nimbus node) and working node (Supervisor node), one worker node running one or more worker processes, worker is a subset of topology, Topology corresponds to one or more worker.

Topology is mainly composed of worker, Executor, task, topology corresponds to one or more workers (is a separate JVM process), and there are multiple Executor threads under the worker. Executor corresponds to one or more tasks, by default a executor corresponds to a task (Spout\bolt), and these tasks are the same Spout\bolt component.

Official

When we create the topology, we configure the number of workers, the number of executor, the number of tasks, that is, the degree of parallelism, improve the degree of parallelism of the topology, can improve the computing power of the topology.

Description

Number of worker: Conf.setnumworkers (num)/config.topology_workers

Number of Executor: Builder.setbolt ()/builder.setspout ()

Number of tasks: Builder.setbolt (). Setnumtasks (Val)

We use the official example to further understand:

Code:

  Config conf = new config ();
  Conf.setnumworkers (2); Use of the worker processes
  topologybuilder.setspout ("Blue-spout", New Bluespout (), 2);//Set parallelism hint to 2
  topologybuilder.setbolt ("Green-bolt", New Greenbolt (), 2). Setnumtasks (4). shufflegrouping ("Blue-spout");
  Topologybuilder.setbolt ("Yellow-bolt", New Yellowbolt (), 6). Shufflegrouping ("Green-bolt");
  Stormsubmitter.submittopology ("Mytopology", Conf,topologybuilder.createtopology ());

Description

1) This topology has two worker processes (Conf.setnumworkers (2)).

2) Bluespout has two executor threads, and each executor corresponds to a task.

3) Greenbolt has two executor threads, each executor corresponding to two tasks.

4) Yellowbolt has six executor threads, and each executor corresponds to a task.

The total degree of parallelism for this topology is 2 + 2 + 6 = 10, which is assigned to each worker with 10/2 = 5 executor.

Bluespout degree of parallelism is 2, Greenbolt degree of parallelism is 2, Yellowbolt is six, improve the degree of parallelism of topology, can improve the computing ability of topology.

With the Storm's demo example, we can see in the Storm UI the topology of two worker,executors, the parallelism of each bolt\spout information