Batch Processing and real-time processing

========================================================== ======================================

Scenario:

Report task: Not a real-time report, data needs to be processed

Recommendation System: requires real-time

Currently, I do not have much knowledge about real-time systems and have little development scenarios. Therefore, I only want to learn

========================================================== ======================================

Distributed System: one system is deployed in multiple sets, multiple storage sets, multiple caches, and multiple databases (RDBMS, nosql )----The cluster system has a large load.

Non-distributed system: it is not just a server, a database, or a cache server

For the high availability of the system, you can use the database system design ideas to make it have a transaction mechanism (Retry Mechanism, commit mechanism, skip mechanism, recovery mechanism)

MANAGER: allocates and schedules resources. It is an idea used to simulate and display the world in software design (tracker, mannager, lanuch, worker, and context)

========================================================== ======================================

Batch Processing: collects a certain amount of business data for unified processing. For example, report data, and monthly statistics on the data for the current month --Scheduled task or batch processing framework (Spring Batch)Hadoop

Real-time processing: Once data is generated, it can be processed.Message Queue (listener and subscription)Storm

Note: These are applicable to scenarios with extremely large data volumes.

However, distributed processing may be a little troublesome, so some frameworks have emerged: hadoop, storm, etc.

Generally, real-time processing uses the event mechanism (CEP)

========================================================== ======================================

Hadoop: a popular term, a distributed Batch Processing Framework

Storm: A recent real-time processing framework

========================================================== ======================================

The thought of concatenating and transforming or splitting:

Split asynchronous execution and merge serial execution

 

========================================================== ======================================