Introduction to Twitter Storm

Background

The past decade has been a decade of Data Processing revolution. MapReduce, Hadoop, and related technologies allow us to process much larger data volumes than before. However, these data processing technologies are not real-time systems-they are not designed for real-time computing. There is no way to simply turn hadoop into a real-time computing system. There are essential differences between real-time data processing systems and batch data processing systems.

However, large-scale real-time data processing has increasingly become a business requirement, and the lack of a "real-time version of hadoop" has become a huge gap in the entire ecosystem of data processing.

Storm fills this gap.

Before Storm appears, you may need to manually maintain a real-time processing network composed of message queues and message handlers. The message handler extracts a message from the message queue for processing and updates the database, send messages to other queues. Unfortunately, this method has the following drawbacks:

1. monotonous: you spend most of your development time configuring where to send messages, deploy message handlers, and deploy intermediate message nodes-you spend most of your time designing and configuring this data processing framework, the message processing logic you really care about accounts for a small proportion of your code.

2. Weak: it is not robust enough. You need to write your own code to ensure that all message handlers and message queues run normally.

3. Poor Scalability: when the message volume of a message processor reaches the threshold, you need to distribute the data. You need to configure these new processors to allow them to process the distributed messages.

For a large number of message processing systems, the final decomposition is the combination of message queues and message handlers, and message processing is undoubtedly the basis of real-time computing. Now the question is: how can we do this without losing data? It can be well extended to a larger message volume and very easy to operate?

Storm meets your needs.

Why is Storm important?

Storm defines a batch of real-time computing primitives. Just as hadoop greatly simplifies parallel batch data processing, storm's primitives greatly simplify parallel real-time data processing. Storm has the following key features:

1. widely used scenarios: storm can be used to process messages and update databases (message stream processing), continuously query a data volume, and return to the client (continuous computing ), perform Real-Time Parallel Processing (distributed method call) on resource-consuming queries. These basic primitives of storm can meet a large number of scenarios.

2. high scalability: the scalability of Storm allows storm to process a large amount of messages per second. To expand a real-time computing task, all you need to do is add the machine and increase the parallel degree setting (parallelism setting) of the computing task ). As an example of Storm scalability, a Storm application processes 1000000 messages per second in a cluster with 10 nodes-including more than one hundred database calls per second. Storm uses ZooKeeper to coordinate various configurations in the cluster so that the Storm cluster can be easily expanded.

3. Ensure no data loss: the real-time system must ensure that all data is processed successfully. The applicable scenarios of systems that will lose data are very narrow, while storm ensures that each message is processed, which is a huge contrast with S4.

4. Exceptional Robustness: Unlike Hadoop, which is notoriously difficult to manage, storm clusters are very easy to manage. Ease of management is one of storm's design goals.

5. Good Fault Tolerance: if some exceptions occur during message processing, storm will reschedule the processing logic for this problem. Storm ensures that a processing logic always runs unless you explicitly kill the processing logic.

6. Language independence: robustness and scalability should not be limited to one platform. Storm's topology and message processing components can be defined in any language, which allows anyone to use storm.

Recommended reading:

Twitter Storm installation configuration (cluster) Notes

Install a Twitter Storm Cluster

Notes on installing and configuring Twitter Storm (standalone version)

Storm practice and Example 1