Architecture Analysis-databus produced by LinkedIn

From: http://blog.sina.com.cn/s/blog_6c9b83990101bbzq.html

1. About databus
LinkedIn opened up its low-Latency Change Data Capture System databus in February 26. This system can capture data on MySQL and Oracle Data sources. Currently, LinkedIn only opens up the connector on Oracle. Databus, as a consistency assurance component in the LinkedIn ecosystem, remains highly effective with low latency. Its biggest feature is its unlimited K Restoration Capability and rich data deep processing capabilities.

Ii. Database-based Data System Architecture

The data system has limited functional segmentation. Traditional databases are used for simple data storage, effectively ensuring data consistency (including transactions); Through millisecond-level event forwarding, indexes or multiple data copies can be built based on business needs to support complex business needs. logs of sequential data change events provide unlimited data recovery capabilities for the entire data system.

Iii. Composition of databus System
Databus relays1: captures database row changes and packs them as data change events in the memory. 2. Listens to clients requests and sends data change events to the client. Databus clients1, check the data change events, and call back the business logic. 2. If the events fail, read all the data change events after the events fail from the databus Bootstrap Server. 3. Add the new client, read data from databus bootstrapserver and then process recent data change events. 4. One client can process all event streams, or multiple clients can process event streams in blocks. Databus Bootstrap producers1 and bootstrap producer are special clients; 2. Receive new data change events from relay; 3. store these events (such as MySQL ); 4. The stored events are used to join the new client or restore the events after the client fails. Databus Bootstrap servers1 listens to client requests and sends stored events to the client. Project address: https://github.com/linkedin/databus