Storm is not fully suited to all real-time applications

Storm is not fully suited to all real-time applications

The rapid rise of streaming data has brought about a new type of application development technology. To cope with the increasing data (such as the large amount of data produced by the Internet of Things and Machine Communication) and improve the online user experience using real-time personalization technology, more and more application development has introduced stream data technology. Streaming Data Applications are defined in a variety of ways, but they all contain three basic functions: Real-time Data Collection, real-time analysis, and automated decision-making.

Stream processing solutions can also take multiple forms. Generally, the architecture includes tuple-at-a-time stream processing tools (also known as complex event processing tools) and batch-oriented stream processing technologies, it is a compromise between flexibility and robustness. The memory database emphasizes the real-time business. Although these methods can solve many problems, these methods vary greatly in terms of development volume, maintenance overhead, storage capacity, and performance. It is important to know how many components are needed to implement the functions you need. The workload for developing, testing, and operating a system depends on the number of components to be integrated.

For example, Apache Storm is one of the most popular tools for developers to use streaming applications. However, a stream processing solution using storm usually requires not only storm. As I mentioned in the previous article, Storm usually uses Kafka for data collection, while Storm and Kafka both require Zookeeper for State management. Finally, as discussed below, the fourth system is required for State management. After all the work is completed, a Storm cluster with three nodes (computing nodes) will easily consume 12 nodes!

Apache Storm and quick data applications

Apache Storm is a stream data processing framework that can run user code snippets on top of data in near real time. It is actually a series of connected pipelines. Essentially, when developers run code in the middle part (Storm MPs Queue), they connect the input data source and backend data storage. Storm latency is closer to CEP (complex event processing) than Hadoop-based systems ). It provides the ability to process events by writing simple code and hand over more background to the framework for processing. It is very attractive for developers who have time to repair and supplement.

Storm is usually used for simple analysis tasks, such as computing and cleaning, to normalize it, and to prepare to consume data for long-term storage. However, Storm cannot perform stateful operations, which is important for real-time decision-making. Status data analysis is at the center of a fast data application, such as personalization, customer engagement, recommendation engine, alarm, authorization, and policy execution. No additional components such as Zookeeper and Cassandra are required. Storm cannot search for dimension data, update summary, or directly act on an event (that is, make real-time decisions ).

These instances are part of a wider class for "fast" data applications. Developers are aware of the value they recognize and act on real-time data, and quick data tools are enriching the value of simple streaming media methods and traditional analysis stores. What is the common topic of fast applications?

First, they all need to read data, such as log records, sensor records, financial bills, click streams, online behavior triggering, and so on. The data is faster than the water sprayed by the fire brigade. The rapid data processing system must be able to read the data smoothly.

Then parse the read data in real time. Before that, some systems can read a stream and submit it to the OLAP system for processing. Such processing may take a long time and cannot take effect in real time. The Quick data processing system uses the event model to process read data and pull existing data (such as dimension tables or historical tables) for processing. The system can display the information or send a reminder so that relevant personnel can make decisions; but it is more appropriate to let the system automatically handle it-for example, "when X-type users meet Y conditions, execute policy Z ".

Finally, quick data processing needs to be integrated with big data. We have a basic assumption that data has a lifecycle. Real-time data is handed over to the quick data processing system for processing, while deep analysis such as machine learning, regression testing, and history query is handed over to OLAP or Hadoop/HDFS-based systems. In this case, the quick data processing system not only submits the expanded data to the deep analysis system, but also receives the processing results of the deep analysis system, such as a rule generated by the rule learning engine.

To understand the role of Storm in the entire fast data processing system, the simplest thing is to see what problems it can handle and what problems it can cause to developers. The design goal of the Storm framework is to submit data from the data source to the user for processing in a way that can freely scale horizontally and has certain error tolerance. It can process the input data at most once or at least once. If the processing fails, it can try again.

The recognition of the persistence status is obviously insufficient. For various reasons, high-speed data applications need to record the status. Retained the previously seen tuples, which can be unprocessed raw data or aggregated data, to help the processing engine find the hidden modes in the data. Before any analysis, obtaining more dimensional data helps enrich these tuples for more extensive generalization and analysis. Generally, final decisions depend on these persistent storage dynamic rules.

Storm makes a mistake on this issue. It allows developers to manage their own access to the persistent state. There are two methods to access the persistent state in Storm. First, in Storm, each working process can store its own state by adding a local third-party storage. Memcached and Redis are now very popular local storage solutions. With the addition of local storage, the data tuples in each State can be used to ignore the previous data tuples, so that the entire working process becomes clear at a glance. However, this self-built method has a negative effect: it cannot be integrated into Storm's fault-tolerant code mechanism. Storm must restart the working process on a running machine and migrate the data tuples from the previous job to the new working process, to handle failed processes or hardware problems. This migration does not contain any local state, whether it is resident in this process or in a system like Memcached. All context information in the failed job will be lost in the new work process.

In addition, it is difficult or even impossible to directly query distributed states distributed across all working processes. If each process wants to aggregate statistics all the time, these statistics must be pushed to another system and queried using powerful query tools. It may be feasible to use Storm's distributed RPC (Remote process call) function to directly capture some data. However, compared with the query functions of today's SQL and NoSQL data storage technologies, these data acquisition functions are strictly limited.

The second method of access status is to connect Storm worker processes to a centralized or distributed persistent storage, such as Cassandra or HBase. Using this method, Storm's processing code can access status information at the GB or TB level, and can easily and independently query persistently stored data from the Storm system. This solves the above problem of Local Storage: The status cannot be rebuilt after failed processes are awakened, and the storage may support more powerful queries in the future. Although this method solves these problems, it greatly increases complexity and brings about reliability issues.

It is worth noting that the combination of Storm and separated storage may weaken the performance. Processing tuples in the memory can easily reach an astonishing speed. Indexes are used to maintain the storage to facilitate query of redundant states, while ensuring consistency as much as possible. The Storm system will be constrained by the performance of distributed storage. If you use external storage to query each tuples, a large amount of network latency may occur. Hiding this latency requires more parallelism and more management, which means more complexity. Now you may have two systems that may fail.

When distributed storage is unavailable, Storm's workflow needs to be processed, which undoubtedly increases complexity. If this is possible, even if there is no complex user code, when using at least one semantics in Storm, developers also need to use at least one semantics in distributed storage.

SQL expansion and quick Data Processing

In today's storage, ACID-compatible, SQL relational databases can be simpler and easier to run applications that previously could not run than Sorm and other products that can replace. ACID transaction relational databases are required to simplify data acquisition at most once or at least once. Because the operation is simple, either completely completed or rolled back.

Most relational databases allow database hosting and transaction logic: Some may even use java code, such as Storm. Therefore, Storm provides more logic and flexibility in relational systems. However, unlike Storm, in a relational database with direct and well-arranged path state data, the hybrid Storm and a distributed or partitioned persistent storage are eliminated.

Now that we have a database storage system like PostgreSQL, why does Nathan Marz create Storm? Because PostgreSQL and its branches are neither horizontally scalable nor capable of processing the throughput of tuples as Storm does. However, in the past five years, some new projects have entered the transactional storage field and are committed to horizontal scaling, fault tolerance, and raw throughput. These system solutions have the same scalability and fault tolerance as Storm, and provide powerful transaction semantic processing functions in combination with direct State operations during processing.

In short, these new platforms provide Storm processing processes and logic, Cassandra's State capabilities, and Kafka's uptake semantics. More importantly, a distributed, transactional (ACID) SQL relational database can process thousands of input transactions or application requests per second.

For applications with rapid data growth, developers are still looking for a stream processing platform that is more complete than Storm and other stream processing solutions. Storm requires a partner database to process any analysis content it produces. Compared with SQL databases of a certain scale, developers can directly compute, store, and perform real-time query and analysis in the database.

Filters in the Storm data extraction pipeline, enrich and group input event streams all need to access the search data and additional input event sources. Storm requires partner databases to store and search data. It does not meet the needs of developers for rapid growth of data applications. A distributed, memory-type relational database provides a set of simpler, more powerful, and more interactive solutions for rapid data growth scenarios.

John Hugg, senior architect of VoltDB, devoted his career to database and information management. As the first engineer for VoltDB productization, he contacted mit scholars, Yale University team, H-Store, And VoltDB research prototypes. John also helped VoltDB build a world-class engineering team to continue developing open-source and commercial products.

Storm Process Communication Mechanism Analysis

Apache Storm History and Lessons

For details about Apache Storm, click here
Apache Storm: click here

Original ENGLISH: Beyond Storm for streaming data applications

This article permanently updates the link address: