Thoughts on a scalable real-time data processing architecture based on Storm

Thoughts on a scalable real-time data processing architecture based on Storm
Problem Introduction

Storm can be used to easily build a cluster-based data framework and define topo to implement business logic.

However, topo has a disadvantage. The topo processing capability comes from the number of workers set during startup. In many cases, we need to be able to adjust the cluster processing capability according to the Business pressure, at this time, a single topo cannot solve this problem.

In order to be able to define processing capabilities more flexibly, you can consider splitting the original topo based on the business domain to achieve non-interference and flexible control. In addition, in order to make more economical use of processing resources, the concept of worker resource pool can be introduced to make full use of resources.

However, this multi-topo architecture has a fatal problem. The topo in storm is independent of each other and cannot communicate directly. Therefore, resource competition may occur when obtaining some key resources. There are two solutions to this scenario:

1. Use distributed locks provided by zookeeper to control key resources. The disadvantage is that there are problems with reliability and efficiency, and scenarios with low processing efficiency requirements.

Second, a third party allocates key resources to avoid competition for resources from the topo itself. This solution introduces a new architecture and increases the complexity of the system.

 

Processing Architecture

The cluster has the advantages of scalability, but it may cause problems such as data synchronization, development and maintenance complexity, and data consistency.

Although we already have a lot of cluster processing frameworks and corresponding components to simplify the development and maintenance workload, from the perspective of project development, we still need to deal with some difficult issues that are not included by mature components.

Storm-defined clusters can provide convenient and scalable processing capabilities. In the entire cluster, topo is equivalent and cannot communicate with each other within the storm runtime environment.

Back to the problem above, through storm, we get real-time cluster processing capabilities; through topo, We can customize the business and facilitate distribution in the node; through changes in the number of worker, you can adjust the processing capability.

With the help of Hadoop and other big data storage platforms and Redis caches and the distributed locks consisting of zookeeper, you can basically build a set of real-time and scalable big data processing platforms.

Component Diagram

 

Multi-top Initialization

The following is a storm-based Class View for multi-topology initialization:

 

Key points and consideration of cache policies

Because it is a real-time data processing platform, it has requirements on efficiency, and access to database storage is often called a bottleneck. Therefore, cache is designed here, And Redis is widely used and stable, there are also mature cache Construction Policies in the industry.

Distributed Lock

Distributed locks are crucial, especially if multiple topo exist in the storm cluster, it is very likely that there will be competition for key resources.

Building distributed locks using zookeeper has already become an application. However, distributed locks built using zookeeper must also have robustness limitations and lock efficiency issues, which need to be considered during design.

Collaboration between Hadoop and Oracle

These two components are very different in terms of cost and use scenario.

In applications, Hadoop can be used to store unstructured data, such as original results. Because of Oracle's huge advantages in storage structure, reliability, and ease of use, you can choose to store the final processing results in Oracle for maintenance and display.

How does Storm allocate tasks and load balancing?

Storm Process Communication Mechanism Analysis

Apache Storm History and Lessons

For details about Apache Storm, click here
Apache Storm: click here

This article permanently updates the link address: