Storm degree of parallelism

1. Storm Parallelism Related concepts

Storm clusters have many nodes, grouped by type Nimbus (Master node), supervisor (from node), and in conf/ STORM.YAML is configured in a supervisor with multiple slots (supervisor.slots.ports), each slot is a JVM, is a worker, in each worker can run multiple threads called executor, in executor run a A component (spout, Bolt) of topology is called a task.

1. Degree of parallelism

Storm's parallelism is done by a lot of supervisor.

Storm's supervisor is running the Spout/bolt task in topology.

A task is the smallest running unit in storm that calculates, representing the running instance of spout and bolts.

The maximum granularity of the execution of a program is the running unit of the process. In supervisor, the process that runs a task is called a worker.

Very many workers can be run on the Supervisor node.

Threads can be run in a worker, which is called Executor. In executor, run the task.

To summarize, Supervisor (node) >worker (process) >executor (thread) >task (instance)

2. Configuration

The 2.1 supervisor is a storm cluster configuration that produces a supervisor node when the Storm supervisor is executed.

The 2.2 worker process is configured in the Storm/conf/storm.yaml file in the options supervisor.slots.ports .

The number of worker processes can also be set by config.setnumworkers (workers) .

2.3 Executor is through builder.setspout (ID, spout, parallelism_hint) and Builder.setbolt (ID, Bolt, parallelism_ Hint) is set.

The 2.4 task is set by boltdeclarer.setnumtasks (num) .

By default, each supervisor starts 4 workers, each worker starts 1 executor, and 1 tasks in each executor.

For example:

123	topologyBuilder.setBolt("green-bolt", newGreenBolt(), 2)               .setNumTasks(4)               .shuffleGrouping("blue-spout);

The above specifies that running Green-bolt,storm using 2 executor, 4 tasks will use 2 executor, each executor run 2 tasks to run Green-bolt.

3. Storm provides rebalance commands to dynamically adjust the degree of parallelism while the topology is running.

For example:

1	$ storm rebalance mytopology -n 5 -e blue-spout=3 -e yellow-bolt=10

You can also perform rebalance on the Web UI.

Storm degree of parallelism