Storm-considerations during use

Storm-considerations during use

Introduction

Over the past few days, in order to optimize the original data processing framework, we have systematically learned some of storm's content and sorted out our experiences.

1. storm provides a data processing idea and does not provide specific solutions.

The core of storm is the definition of topo, and topo carries all the business logic. We organize solutions based on storm's topo to orchestrate private business implementation logic.

Storm's business abstraction is the most basic information stream processing paradigm.

2. storm supports multiple languages

The processing framework provided by storm supports multiple languages. java is only one of them. We can use different languages to implement topo.

3. storm will break the dependent class into the same class

This may be strange for beginners, but it is a compromise made by storm for distributed deployment. If the jar package itself carries everything required during the runtime, storm itself can only exist as a runtime framework, and the distribution of jar packages between clusters will become much easier.

4. storm supports multiple topo

As mentioned above, storm itself is only a data processing framework, and the core of business logic is topo, we can submit multiple topo to the storm cluster to make full use of storm's cluster processing capabilities. Because the configuration is effective for the topo, you can also define the topo processing capability.

5. Compared with Hadoop, storm is better at real-time processing.

Storm does not have complicated and lengthy hadoop pre-processing processes. Distributed storm and distributed hadoop are a good partner who can work together to form a Big Data Processing framework.

6. storm's topo communication is basically not feasible.

Topo runs independently in storm and does not interfere with each other. Direct topo communication at the code level is not feasible (at least I did not find it ), only storm ui and limited APIs can be used for monitoring.

If there is a communication requirement, you can consider using zookeeper or database as the communication intermediary, and using topo to communicate indirectly with the intermediary.

Thoughts on a scalable real-time data processing architecture based on Storm

How does Storm allocate tasks and load balancing?

Storm Process Communication Mechanism Analysis

Apache Storm History and Lessons

For details about Apache Storm, click here
Apache Storm: click here

This article permanently updates the link address: