Introduction to real-time data stream processing

Recently in the group made a simple sharing, the topic is the real-time data stream processing, the main point is the recommendation system in a variety of real-time data stream real-time computing problems, of course, such as search trends, advertising real-time billing and so may encounter similar problems, PPT inside a simple introduction of the system on the actual structure and flow, System on-line running nearly half a year, although there are no problems, some of which are worth revising and improving, if there are business-related requirements, these problems will be improved.

main issues to be addressed:

1. The problem of chaos, this in most current real-time system is more common, that a variety of data to reach the system time is inconsistent, the traditional window join method is not too work

2. Low latency and scalability, real-time data processing to serve online model and service, second-level latency is a must, in order to cope with possible data growth, scalability is also necessary.

Existing systems are facing problems:

1. Data service problem, real-time computing processing intermediate data is not very good for third-party services to share, need to have intermediate data landing or API basic data exposure interface, to avoid duplication of computation and processing

2. The problem of data processing efficiency, message accumulation, cache processing, etc. when pulling data from Kafka

3. Cache processing problems, including service fault tolerance processing and caching failure mechanism, fault-tolerant processing mainly for the service in the cache processing exceptions and failure retry service, cache active invalidation and so on.

PPT behind the possible several scenarios, the individual in Kafka consumer pull data when the Ratelimiter, caching and data service is trying, not necessarily the right solution, relatively simple, effective method is the most practical, need to choose according to the application scenario.

PPT Share Address

Introduction to real-time data stream processing