Learn Storm_storm basic concepts with me

First, let's look at the basic concepts in storm with a comparison of Storm and Hadoop.

　　

Next, let's look at these concepts in more detail.

• Nimbus: Responsible for resource allocation and task scheduling.
• Supervisor: Responsible for accepting tasks assigned by Nimbus, starting and stopping worker processes that belong to their own management.
• Worker: A process that runs a specific processing component logic.
• Each Spout/bolt thread in the Task:worker is called a task. After Storm0.8, the task is no longer corresponding to the physical thread, and the same Spout/bolt task may share a physical thread called executor.

The following diagram depicts the relationships between the above roles.

　　

Figure 1.2 Storm inter-role relationships

• A real-time application running in Topology:storm because the flow of messages between components forms a logical topological structure.
• Spout: The component that produces the source data stream in a topology. Typically, spout reads data from an external data source and then translates it into the source data inside the topology. Spout is an active role with a nexttuple () function in its interface, and the storm framework calls this function continuously, so that the user can generate the source data in it.
• Bolt: A component that accepts data in a topology and then executes the processing. Bolts can perform any operation such as filtering, function manipulation, merging, writing the database, and so on. Bolt is a passive role with an execute (Tuple input) function in its interface, which is called when the message is received, where the user can perform the action he or she wants.
• Tuple: The basic unit of a single message delivery. It was supposed to be a key-value map, but because the field names of the tuple passed between the components were already defined beforehand, the tuple would simply fill in each value in order, so it would be a value list.
• Stream: A stream is formed by a stream of tuples.

Hadoop is the idea of implementing MapReduce, which computes data slices to process large amounts of offline data. The data processed by Hadoop must be in a database that is already stored in HDFs or similar to HBase, so Hadoop is implemented by moving computing to these data-storing machines to improve efficiency, and storm is a flow computing framework, The processed data is in the real-time message queue, so we need to write a topology logic, and receive incoming data to deal with, so the mobile data is distributed evenly to the machine resources to achieve high efficiency.

The advantage of Hadoop is that it handles large volumes of data (bottlenecks are hard drives and namenode, networks, etc.), and is flexible to analyze data flexibly by stitching Hadoop commands such as DSL,MDX, or directly using Hive,pig. Adapt to the combination analysis of a large number of dimensions. The disadvantage is slow: to distribute the jar package before each execution, Hadoop writes the data to the local file system each time the map data exceeds the threshold, and then reads it again at reduce.

The advantage of storm is that full-memory computing, because memory addressing speed is more than millions of the hard disk, so storm speed compared to Hadoop very fast (bottleneck is memory, CPU). Its disadvantage is not enough flexibility: must first write good topology structure to wait for data in analysis.

Storm is concerned with data processing one write at a time, while Hadoop is concerned with data writes once and multiple queries are used. The storm system runs continuously, and Hadoop often just calls the data when the business needs it.

Learn Storm_storm basic concepts with me