A low-latency distributed database synchronization system--databus

Every time I saw the big brand of Motorola across the street, we thought about Google's $12.5 billion acquisition of Motorola Mobility, and then sold it to Lenovo for $2.9 billion. Google has made decisions that take long-term interest into account, and in this deal, Google has acquired Motorola's most valuable thousands of patents, a robust Android ecosystem. Google has built a solar power station in a region that occupies a huge area of the United States, calling it a "baked bird" project. Because the birds will fly from the sky because it is too hot to fall from heaven to become mature birds. The electricity generated is only enough for a small village to use. But it is this willingness to try to promote the development of science and technology. Let Google for so many years stand up.

Databus is a low latency Distributed database synchronization system (a source-agnostic distributed change data capture system) from LinkedIn Open source, which provides reliable data capture, flow, and data processing capabilities.

Application Scenarios:

The main application scenario: Database data subscription between the departments or across the computer room. Similar products have Ali's canal. Supports Oracle and MySQL databases.

Databus and canal, data is based on database log mining. My previous blog also mentions that today's data processing can be broadly divided into two categories: online transaction processing OLTP (line Transaction processing) and online analytics processing OLAP (Online analytical Processing).

OLTP is the main application of the traditional relational database, mainly basic, daily transaction processing, such as bank transaction. But what is commonly referred to as OLTP is the database engine component that is integrated into SQL Server because Microsoft has a white paper, SQL Server in-memory OLTP Internals Overview. But OLTP can actually be called: transaction-oriented real-time processing system. The performance indicator is TPS.

OLAP is a solution for fast response Multidimensional analysis. The data source that is oriented is the Data warehouse. Support complex analysis operations, focus on decision support, and provide intuitive and understandable query results.

Come back and say the most important feature of Databus,databus is its infinite backtracking function. When a consumer's full data copy, or the consumer's existing data and the main OLTP database is very different, can be low-latency synchronization.

　　

Core components:

The core component of Databus is the Databus Relay. Mainly responsible for two jobs:

1> reads the changed rows from the Databus source database and serializes the event stream into memory.

The 2> accepts the client's request and returns the data change event stream to the client.

Implementation principle:

Databus uses Open-replocator as the component to get database change events. The function is to get the Binlog log of MySQL in real time and encapsulate it as an event object and then call the registered listener for processing.

Technical framework:

• Event Producer: A Change event used to read a database, converted to a Avro type and stored in memory.

Let's talk about Avro. Apache Avro is a programming language-independent data serialization system that was created by the father of Hadoop. It provides: Rich data structure types, fast compressible binary data forms, file containers for storing persistent data, remote procedure calls, and simple integration with dynamic languages. There is no need to generate code to read and write data files and to use the RPC protocol, and code generation as an optional optimization is only worth implementing in a statically typed language.

• The Circular Buffer:relay has one or more annular buffer pools that are used to hold the change events in order of the increment system change number (SCN).

Careful readers will find each middleware has its own unique buffer strategy, this buffer strategy is very mature determines the quality of this middleware.

• SCN Writer/reader: Used to read and inhale the SCN number to the hard drive.
• RESTful interface: It exposes a restful interface to push data change events to the client.

Source:

Databus is developed in Java. The GitHub address is: Https://github.com/linkedin/databus.

Off-topic time:

Ever since I graduated from work, I have been able to feel some problems from time to time. For example: When I was anxious, my brow was locked, and I couldn't speak politely. Another example of my way of speaking is also insufficient skill. These seemingly not-so-central issues are now my first priority. Because before there is no matter, I do not manage, I will control myself. So these little problems affect me at best. Many of my decisions now have a direct impact on our team.

Therefore, in the context of the actual scope of control, I would instead make the problem more obvious and make some minor contradictions. Why do this, because I never do "everybody says it's right" thing. I only do the things that I fully accept in my heart.

Everyone who knew me before knew that I had a loud voice. It's a little better now, though sometimes I forget to lower my voice. My family in our town's most prosperous lot, behind is the market. It's noisy 3 o'clock in the morning every day. The window of my house is just right on the market. So I grew up with "again noisy and I do not have the ability, so I am very insensitive to noise." So I took a lot of effort to let my heart completely accept the "This will disturb others, the best volume is not too big, shock to others ears, not too small, others can not hear clearly"

I now need to let myself know what the consequences of my problems are, and there are indeed problems. But it's really a stupid way to do that. A wise man is a man he can foresee, and life can be a lot of detours.

I have been working for 10 years and there is no substantial improvement. At the time of graduation, my degree was almost the same as the company at this level. Everyone, le vision, the new beauty big. There is no level of ascension, but the rank is lower than the same term.

Compared with other people, maybe I went to Japan 1 in a year. He went to Silicon Valley in America. There are also hundreds of patents. I was wondering why none of this would bring me any real benefits.

I do have a strong language skills, but I just want to do technology. Self-willed is to have a price. The price is that language ability is useless.

Going to Japan and Silicon Valley, this is a job opportunity, not what ability. What should have brought me is temperament, conversation. But my temperament and conversation did not go up. That is, I went to the white, did not learn anything. Literacy did not improve.

A patent is a real benefit to the company, and it has no relation to quantity. I did not have a particularly good patent, played a special role.

In general, I am a very ordinary, and 1.5 months on the 31 birthday of the old technology. There is no advantage at all. And, I'm a little impetuous.

Before I go to Le vision, every effort I make takes place when I have to work hard. Before the study, because of worry, do not test how to do it, so hard grades up. The crisis is gone and back to the original. I work hard when I have a job to do. It is my duty to do what I do not finish. Then basically is busy one months two months idle for half a year.

I had a good start when I went to the music vision. Why, because before I went to Le Vision, I went to the muddy waters, of course not at work. But I was very depressed at the time I was happy. Until the end I carved a few scars on their own face, nearly 1 years is good, this period of time I have depressed their own scars, the original because what depressed completely do not remember. So this is my least impetuous period of time. However, I think compared with others, also quite impetuous.

For now, I think the starting point is very low. Suddenly feel that they have nothing. But at least at every stage the things to do are done, the trial of the mistake has been actually tried. I got what I really wanted. Give up, it is not worth their own feel. There is only one thing left, and I want to reach the height I should reach at work. Because it industry is very cruel to age and girls. I don't want to be a name, not a profit, do not love rights, but I would like to work.

Now the company, everyone has a lot of me I am worthy of learning places. Do not follow others to learn, their own growth also needs a lot of improvement. Methodological aspects, architectural aspects. I can not drag a long time to adapt, I need to be able to have some time, every day to write some code, the fundamental things do not want to put down.

I've never liked being called old by others. has always been immature and decent. Sometimes the hair is messy, sometimes without consideration. This is young, I hope others like their original appearance. Now, more and more think about how to do things well. Static son of the article which has Bushon love. Today, I just want to say thank you to the man who met me when I was young and I'm afraid I won't fall in love with anyone who is bent on doing things.

A low-latency distributed database synchronization system--databus