Big Data reading Notes (2)-Streaming calculation

The early and current "streaming computing" systems are called "Continuous query Processing class" and "extensible data Flow Platform Class" computing system respectively.

Features of the streaming computing system:

1) Low latency 2) excellent system fault tolerance 3) very strong system expansion capacity 4) flexible and powerful application logic expression capability

Current typical streaming computing systems:

S4,storm,millwheel,samza,d-stream,hadoop Online,mupd8 and so on.

Among them, storm and Millwheel are more prominent in all aspects.

Streaming Computing System Architecture:

There are two common streaming computing architectures: master-Slave mode and peer-to-peer mode.

Master-slave mode storm, Millwheel, and Samza

Peer Mode S4

Samza is a combination of the message system Kafka and hadoop2.0 's resource management system yarn, the architecture is unique, but essentially follows the master-slave architecture.

Delivery Guarantee

Three possible: Delivery at least once (At_least once delivery), at most once (At_most once delivery), exactly once (exeact-once delivery).

Storm provides a "just-in-time" semantics in the system through the delivery guarantee mechanism and the transactional Topology union.

Samza relies on Kafka's message persistence federated broker caching mechanism to guarantee "just-in-time" semantics.

Big Data reading Notes (2)-Streaming calculation