The core technologies and basic components of storm

The core of the storm framework consists of 7 parts,

Topology (topology)

A topology is the calculation of a graph. The user contains processing logic for each node in a topology, and the connections between nodes show how the data should be passed between nodes. The topology is very simple to run.

Stream (Flow)

Flow is the core abstraction of storm. A stream is an unbounded tuple sequence that can contain integers, long integers, short integers, bytes, characters, double digits, floating-point numbers, booleans, and byte arrays. Users can use custom types in native tuples by customizing the serializer.

Spout (spout)

Spout is the source of the topology stream. Typically spout reads a tuple from an external source and commits it to a topology (such as a kestrel queue or Twitter API). Spout can be classified into two modes: reliable and unreliable. Spout can emit more than one stream.

Bolt (Bolts)

The processing of all data in the topology is done in the bolt. Bolt can complete data filtering, business processing, connection operation, connection, database access and other operations. Bolts can do simple stream transformations, emit more than one stream, and the main method is the Execute method. It is entirely possible to start a new thread in the bolt to do asynchronous processing.

Stream grouping (stream grouping)

Flow grouping defines how the stream should be partitioned in the Bolt's task. Storm has 7 built-in stream grouping interfaces: Random grouping (Shuffle grouping), field grouping (Fields grouping), all grouping (all grouping), global grouping (globally grouping), no grouping (none grouping), direct grouping, notebook, or random grouping (Local grouping or shuffle grouping).

Task (Tasks)

Each spout or bolt performs many tasks in the cluster. Each task corresponds to the execution of one thread, and the Liu Group defines how to send a tuple from one task set to another. You can set the degree of parallelism for each spout or bolt by using the Setspout () and Setbolt () methods of the Topologybuilder class.

Worker (worker process)

Topology process execution across one or more worker nodes. Each worker node's process is a physical JVM and topology performs a subset of all tasks.

The core technologies and basic components of storm