Original:http://highlyscalable.wordpress.com/2013/08/20/in-stream-big-data-processing/
Ilya Katsov
For quite some time since. The big data community has generally recognized the inadequacy of batch data processing.
Very many applications have an urgent need for real-time query and streaming processing. In recent years, driven by this concept. Has spawned a series of solutions. Twitter Storm,yahoo S4,cloudera Impala,apache Spark and Apache Tez have added big data and a NoSQL camp. This paper attempts to explore the techniques used in streaming systems and analyze their relationship with large-scale batch processing and OLTP/OLAP databases. and explore how a unified query engine can support streaming, batch, and OLAP processing at the same time.
In Grid Dynamics, we are faced with the need to build a streaming data-processing system. You need to handle 8 billion of events every day and provide fault tolerance and strict transactional, that is, you cannot lose or repeatedly handle events.
The new system is a complement and successor to the existing system. Existing systems are based on Hadoop, data processing is high latency, and maintenance costs are too high.
Such requirements and systems are quite generic and typical. So we describe it as a normative model, as an abstract problem statement.
A high-level presentation of our Production environment Overview:
watermark/2/text/ahr0cdovl2jsb2cuy3nkbi5uzxqvawrvbnr3yw50b2jl/font/5a6l5l2t/fontsize/400/fill/i0jbqkfcma==/ Dissolve/70/gravity/center ">
This is a typical big data infrastructure: Every application in multiple data centers is producing data, and data is delivered through the subsystem to HDFS on the central facility, and the raw data is aggregated and analyzed through the standard Hadoop stack (mapreduce,pig,hive). The aggregated results are stored on HDFS and NoSQL, and then exported to the OLAP database for customized user application access. Our goal is to equip the entire facility with a new streaming engine (see bottom of the figure). To handle most dense data streams. Transport pre-aggregated data to HDFS, reducing the amount of raw data in Hadoop and the load on the batch job.
The design of the streaming engine is driven by the following requirements:
The engine can also handle the relative static data (admixtures) that is loaded from the aggregated data.
More complex multi-track data mining algorithms are not within the short-term target range.
Pipelines can also be stateful. The middleware of the engine provides persistent storage to enable the stateful checkpoint mechanism.
All of these topics will be discussed in the section of this article.
To figure out how to implement such a system, we discuss the following topics:
In addition, we have researched the emerging technologies and provided some implementation tips.
The article isbased on a in the project developed at Grid Dynamics Labs. Much of the Creditgoes to Alexey Kharlamov and Rafael Bagmanov who led the project and Othercontributors:dmitry Suslov, K Onstantine Golikov, Evelina Stepanova, Anatolyvinogradov, Roman belous, and Varvara Strizhkova.
Distributed streaming data processing is obviously associated with a distributed relational database. Many standard query processing techniques can be applied to the streaming engine, so it is very useful to understand the classical algorithms of distributed query processing and to understand their relationship with streaming and other popular frameworks such as MapReduce.
Distributed query processing has been developed for decades and is a very large area of knowledge. We start with a concise overview of some of the major technologies and provide the basis for the discussion below.
Distributed and parallel query processing is heavily dependent on data partitioning, breaking up big data into multiple slices so that individual processes are processed separately.
Query processing may contain multiple steps. Each step has its own partitioning strategy. So the data shuffling operation is widely used in distributed database.
While the optimal partitioning for selection and projection operations requires some skill (for example, in a range query), we are able to use hash partitioning to distribute data across processors in a streaming data filter.
Distributed joins are not so simple and need to be studied in depth. In a distributed environment. Parallel joins are implemented through data partitioning, that is, data is distributed across processors, and each processor executes a serial join algorithm (such as nested loops join or sort-merge join or hash join) to process part of the data. Finally, the results are obtained from different processors.
The distributed join mainly uses two kinds of data partitioning techniques:
Disjoint data partitioning technology uses join key to shuffle data to different partitions, and partition data does not overlap.
Each processor runs a join operation on its own partition data, resulting in a simple concatenation of different processor results. Consider examples of R and S DataSet joins. They are join with numeric key K and are partitioned with simple modulo functions.
(If the data is based on a certain policy.) Already distributed on each processor):
Demonstrates the partition-broadcast join algorithm. The dataset R is divided into disjoint partitions (R1,R2 and R3 in the figure). The data set S is copied to all processors. In the distributed database. The partitioning operation itself is usually not included in the query process, since data initialization is already distributed across different nodes.
Such a strategy applies to large dataset join small datasets, or joins between two small datasets. Streaming data processing systems can apply such techniques, such as the use of static (admixture) and data flow to join.
Group by The process also relies on shuffling . is essentially similar to MapReduce. Consider the following scenario, where the dataset is grouped by character field, and then the numeric field for each group is summed:
In this example. The calculation process consists of two steps: Local rollup and global rollup. This basic and map/reduce operation corresponds. The local rollup is optional. Raw data can be transferred, shuffle, summarized in the global rollup phase.
The general view of this section is that. The algorithms above are naturally implemented through the message-passing architecture pattern. In other words, the query run engine can be seen as a distributed network node that is connected by Message Queuing. Conceptually and streaming pipelines are similar.
In the previous section. We note that very many distributed query processing algorithms are similar to messaging networks.
But. This is not enough to illustrate the efficiency of streaming: all operations in a query should form a link, and the data flow is smoothed through the entire pipeline, that is. No matter what the operation can not block the processing process, can not wait for a large input without producing no matter what output. There is no need to write intermediate results to the hard disk.
There are some operations such as sorting, natural incompatibility of the idea (it is obvious that the sort does not produce any output before processing the input), but the pipeline algorithm is suitable for very many scenes. A typical pipeline for example with what is seen:
In this example, a three processor hash joins four datasets: R1,s1. S2 and S3. First, parallel to the S1,S2 and S3 set up a hash table, and then R1 the tuple flow through the pipeline, from the S1. Finds matching records in the S2 and S3 hash tables. Streaming is a natural use of this technology to enable the join of data flow and static data.
In relational databases, join operations also use the symmetric hash join algorithm or other advanced variants.
A symmetric hash join is a generalized form of a hash join algorithm. A normal hash join requires at least one input for a fully usable talent output (where one input is used to establish a hash table), and a symmetric hash join maintains a hash table for two inputs when the data tuple arrives. Fill separately:
watermark/2/text/ahr0cdovl2jsb2cuy3nkbi5uzxqvawrvbnr3yw50b2jl/font/5a6l5l2t/fontsize/400/fill/i0jbqkfcma==/ Dissolve/70/gravity/center ">
When a tuple arrives, it looks for the corresponding hash table from another data stream. If a matching record is found, the result is output. The tuple is then inserted into the corresponding hash table of its own data stream.
Of course. Such an algorithm does not make sense to join the infinite stream altogether. Very much under the scenes. Joins are used for a limited time form or for other types of buffers. For example, use LFU to cache the most frequently used tuples in a data stream.
The symmetric hash join applies to the rate at which the buffer is larger than the stream, or if the buffer is frequently cleared by the application logic, or if the cache recycle policy is not predictable. In other cases. Using a simple hash join is sufficient. Because the buffer is always full, it does not clog the processing flow:
It's worth noting that. Streaming processing often requires complex flow association algorithms, record matching is no longer based on the conditions of word multiplies, but based on scoring measures, in such scenarios. There is a need to maintain a more complex cache architecture for two streams.
In the previous section, we discussed some of the standard query techniques used in large-scale parallel stream processing.
Conceptually, it seems that an efficient distributed database query engine is capable of streaming, and vice versa, a streaming system should also be able to act as a distributed database query engine. Shuffling and pipelines are key technologies for distributed query processing, and they can be implemented naturally through messaging networks.
But the real situation is not that simple.
In the database query engine. Reliability is not so critical, because a read-only query can always be executed again, while a streaming system must focus on the reliable handling of messages. In this section, we discuss the techniques for streaming systems to guarantee message delivery. and other patterns that are less typical in standard query processing.
In a streaming system, the ability to turn back the clock and replay the data stream is critical. Because of the following reasons:
As a result, input data is typically streamed from the data source through the buffer to the streaming pipeline, agreeing that the client moves the read pointer back and forth in the buffer.
The Kafka Message Queuing system implements a buffer that supports scalable distributed deployment and fault tolerance, providing high performance at the same time.
Stream playback requires that the system design consider at least the following requirements:
In a streaming system, events flow through a series of processors until the end point (for example, an external database). Each input event produces a forward graph of the Child event node (blood), with a graph that ends with a final result.
To ensure the reliability of data processing, the entire diagram must be successfully processed, and the process can be restarted in the event of a failure.
Achieving efficient blood tracking is a challenge. Let's start by introducing how Twitter storm tracks messages, guaranteeing "at least one" message processing semantics:
Such a case. Events are associated to multiple initial events, carrying multiple initial IDs (the third part of the picture is a yellow background event).
The framework sends a commit message to the event source node (part Three).
However, the 64-bit long ID is sufficient to guarantee a very low error probability, and the probability of error is probably 2^ (-64), which is acceptable in most applications. The main advantage of the algorithm is that only a small amount of memory is needed to save the signature table.
The above implementations are elegant and have a de-centering feature: Each node sends a confirmation message independently. You do not need a central node to explicitly track blood.
However. Transaction processing becomes more difficult for data streams that maintain a sliding form or other type of buffer. For example, a sliding window may contain thousands of events. A very large number of events are in uncommitted or computational state and need to be persisted frequently, and it is difficult to manage the event confirmation process.
Apache Spark[3] uses the second implementation method, the idea is to think of the result as the input data processing function. In order to simplify the blood tracing, the frame batch processing event, the result is also batch, each batch is the input batch processing function.
The results can be calculated in batches, assuming that a calculation fails, the framework simply needs to run it again. Consider the following example:
In this example, the framework joins two streams on a sliding form. Then the results go through a processing phase. The framework splits the flow into batches, specifying IDs for each batch, and the framework can obtain the corresponding batches at any time by ID.
Streaming is split into a series of transactions, each of which processes a set of input batches, uses the processing function to transform the data, and saves the results. In, the Red highlighting section represents a single transaction. Assuming a transaction fails, the framework re-runs it, and most importantly, the transaction is capable of parallelism.
This approach is simple and powerful, enabling centralized transaction management and providing natural "just-in-time" message processing semantics.
Such techniques are also applied to batch processing and streaming at the same time, as they are split into a series of batches regardless of whether the input data is streaming.
In the previous section, we used the signature (checksum) in the blood tracking algorithm to provide "at least once" messaging semantics.
This technology improves the reliability of the system, but leaves at least two open issues:
The state of the calculation needs to be persisted or replicated to avoid state loss when the node fails.
Twitter Storm uses the following protocols to address these issues:
The transaction ID is monotonically incrementing, for example. The first event ID is 1 and the second batch ID is 2. etc.). Suppose the pipeline fails while processing a batch, and this batch of data is sent again with the same transaction ID.
At last. The framework notifies the transaction that it is finished, and all node commits state changes (such as updating to an external database).
Assuming that the data source is fault-tolerant and can be replayed, the transaction guarantees a "run only" processing semantics. However, even with bulk batching, persistent status updates can lead to severe performance degradation.
So. You should minimize or avoid intermediate calculation result states.
As a supplement, state writing can also be implemented in different ways. The most straightforward way is to copy the in-memory state to the persisted storage during the transaction commit process.
Such a way does not apply to large-scale states (such as sliding forms). Another option is to store a transaction log, such as a series of operations logs that convert the original state into a new state (a set of join and cleanup events for sliding forms). Although this approach requires rebuilding the state from the log, disaster recovery becomes more cumbersome, but in very many scenarios. It can provide better performance.
The scalability of the intermediate and finally calculated results is very important, which can greatly simplify the design, realization, maintenance and recovery of the streaming data processing system. Scalability means that the results of large-scale or large-capacity data partitioning can be combined by a smaller time frame or smaller partitioning results. For example, the daily PV amount equals the sum of PV per hour.
State-additive consent to split the data stream, as we discussed in the previous section, each batch can be independently computed/recalculated, which helps to simplify blood tracing and reduce the complexity of state maintenance.
The implementation of the additive is often not easy:
We have to maintain a list of user IDs. The additive is achieved through the intersection of the ID list and the operation of the set.
The size and processing complexity of the user ID list is equivalent to the original data.
A sketch (sketches) is an efficient way to convert non-additive values to an additive value. In the example above, the ID list can be replaced by a compact additive statistic counter. Counters provide approximate values rather than exact values, but are acceptable in many applications. Sketches are popular in specific areas such as Internet advertising and can be seen as an independent streaming mode. See [5] for an in-depth overview of sketching techniques.
Streaming calculations typically rely on time: summaries and joins are generally useful on sliding time forms, and processing logic often relies on event time intervals. Obviously, the streaming system should have its own time view and should not use CPU wall clock time. Data flow and specific events are replayed as a result of a problem. So it's not easy to achieve the right time tracking.
Usually. The global concept of logical time can be implemented in the following way:
We've discussed how persistent storage can be used for stateful checkpoints, but this is not the only effect of streaming systems introducing external storage. Consider using Cassandra to join multiple data streams on a time form.
Instead of maintaining the in-memory event buffers, we were able to save the incoming events of all the data streams to Casandra, using the join key as the row key and seeing:
watermark/2/text/ahr0cdovl2jsb2cuy3nkbi5uzxqvawrvbnr3yw50b2jl/font/5a6l5l2t/fontsize/400/fill/i0jbqkfcma==/ Dissolve/70/gravity/center ">
On the other side, the second processor periodically iterates through the data records. Cleans up events in the time-out form by assembling and sending records after a join.
Cassandra can also speed up the process by sorting events based on timestamps.
The wrong implementation fall short the entire streaming process--even with high-speed storage systems such as Cassandra or Redis, writing each piece of data alone introduces a serious performance bottleneck.
On the one hand, the use of storage systems provides a more intact state persistence feature. It is assumed that an acceptable performance target can be achieved in a very large number of scenarios by using optimization methods such as batch write.
Streaming data processing often deals with "what is the sum of a data value for the last 10 minutes stream", such as a continuous query on a time form. The most straightforward solution to this type of query is to calculate aggregate functions such as sum for each time form separately. Obviously, such a scheme is not optimal. Because two consecutive time form instances have a high degree of similarity.
Suppose the form of the moment T consists of a sample {s (0), S (1), S (2),..., s (T-1), S (t)}, then the time t+1 form will include {s (1), S (2), S (3) ..., S (t), S (t+1)}. Observe the ability to use incremental processing.
The incremental calculation on the time form is also widely used in digital signal processing. Contains software and hardware. A typical example is the calculation of the sum value.
Assuming that the sum value of the current time form is known, the sum value of the next time form can be obtained by adding a new Swatch and subtracting the oldest sample from the form.
Similar techniques can be used not only for simple aggregation functions such as sum and product, but also for more complex conversion processes. Example. The SDFT (sliding discrete Fourier transform) algorithm [4] is much more efficient than using the FFT (high-speed Fourier transform) algorithm for each form.
Now go back to the practical questions raised at the beginning of the article. We designed and implemented our own streaming system based on Storm,kafka and Cassandra, which applied the technology described earlier. This. We only provide a concise overview of the solution-specific descriptions of the holes and techniques in all the implementations are too lengthy and may require a separate article.
The system uses Kafka0.8 for granted.
Kafka, as a partitioned, fault-tolerant event buffer, enables streaming playback and easy addition of new event producers and consumers. Enhances the scalability of the system. The ability of the Kafka to read pointer backtracking also makes it possible to randomly access the incoming data batches, as appropriate. Able to achieve spark-style lineage tracking. It is also possible to point the system input to HDFS for processing historical data.
As described in the previous description. The Cassandra is used to implement state checkpoints and persist storage aggregations. In very many usage scenarios. The Cassandra is also used to store the results finally.
Twitterstorm is the cornerstone of the system.
All active query processing is performed in Storm's topologies, topologies and Kafka, and Cassandra Interact.
Some data streams are simple: Data arrives kafka;storm read and processed. Then store the results in Cassandra or somewhere else. Other data streams are more complex: A storm topology passes the data to a topology with Kafka or Cassandra. Shows two data streams of this type (red and blue curve arrows).
Existing technologies such as Hive,storm and Impala allow us to handle big data with ease, complex analysis and machine learning using batch processing. Online analysis using real-time query processing, continuous query using streaming processing. Further. The lambda architecture can also effectively integrate these solutions. This brings us to the question of how these technologies and methods can be assembled into a unified solution in the future. In this section we discuss distributed Relational query processing, batch processing, and streaming processing, the most prominent common denominator. Total can cover all user scenarios, thus the most promising solution in this area.
The key point is that relational query processing, MapReduce, and streaming can all be implemented using the same concepts and techniques as shuffling and pipelines. At the same time:
These features are not critical for batch processing where the calculation process is very convenient to restart.
The above two points imply that the adjustable persistence strategy (memory-based messaging or storage on the hard disk) and reliability are notable features of our imagined unified query engine.
The unified query engine provides a set of processing primitives and interfaces for a high-level framework.
In emerging technologies, the following two are worth focusing on:
u=d6ckvfygozxerfxgxs9ubereu7w0hys0r2ltz5sv7gej13itwfjruhelcugrulhgffampeiwsyv%2fvwpsxg0ll4tlhy8n4vvx% 2fhwfceebudaviqeh5yspl2kee4hycg%3d%3d&b=6 ">j Kinley," The Lambda architecture:principles for architecting Realtimebig Data Systems "
U=d6ckv%2fgzsibbwltyymvue15e9rgnv4c0smlpbzj0&b=6 ">http://hortonworks.com/hadoop/tez/
U=d6ckpocktoqcxunfydmzbb9gq77p&b=6 ">http://spark-project.org/
Translate-in-stream Big Data processing streaming large data processing
Start building with 50+ products and up to 12 months usage for Elastic Compute Service
1 on 1 presale consultation
24/7 Technical Support 6 Free Tickets per Quarter Faster Response
Alibaba Cloud offers highly flexible support services tailored to meet your exact needs.
A comprehensive suite of global cloud computing services to power your business
Payment Methods We Support
