The study and application of "reprint" Zookeeper

Introduction

Cloud computing is becoming more and more popular today, the single machine processing capability can not meet our needs, we have to use a large number of service clusters. Service cluster in the process of external services, there are a lot of configuration needs to be updated at any time, the need for coordination between services, how can this information be pushed to each node? and ensure the consistency and reliability of information?

It is well known that distributed coordination services are difficult to implement correctly, and they easily make mistakes in terms of competition and deadlock. How to save energy in this area? Zookeeper is a good choice. The motivation behind zookeeper is to relieve the pain of distributed applications in achieving coordinated services. On the basis of introducing the basic theory of zookeeper, this paper realizes a configuration management center with zookeeper, and uses zookeeper to distribute the configuration information to each service node, and ensure the correctness and consistency of the information.

What is zookeeper?

Citing the official saying: "Zookeeper is a high-performance, distributed, open-source distributed Application coordination Service. It provides simple and original functionality, and distributed applications can be based on it to achieve more advanced services, such as synchronization, configuration management, cluster management, and name space. It is designed to be easy to program, using the file system tree as the data model. The server runs on Java, providing the Java and C client APIs.

Zookeeper overall structure

The zookeeper service itself consists of a cluster (2n+1 service allows N to fail). Zookeeper Service has two roles, one is leader, responsible for writing services and data synchronization, the rest is follower, provide read service, leader after the failure will be in follower to re-elect the new leader.

The zookeeper logical diagram is as follows,

1. The client can connect to each server, and the data for each server is exactly the same.
2. Each follower is connected to the leader and accepts leader data update operations.
3. The server logs transaction logs and snapshots to persistent storage.
4. Most servers are available, and the overall service is available.

Zookeeper Data Model

Zookeeper is represented as a hierarchical file system directory tree structure (unlike file systems, nodes can have their own data, while the directory nodes in the file system have only child nodes).

The data model structure diagram is as follows,

A circular node can contain child nodes, and a polygon node cannot contain child nodes. One node corresponds to an application, and the data stored by the node is the configuration information required by the application.

Zookeeper Features

• Sequential consistency: Updates the data in the order in which the client sends the request.
• Atomicity: The update either succeeds or fails, and no partial updates occur.
• Singleness: You see the same view regardless of which server the client connects to.
• Reliability: Once the data has been updated successfully, it will remain until the new update.
• Timeliness: The client gets up-to-date data within a certain time period.

Zookeeper Application Scenario

• Data publishing and subscription (my business uses this feature, which is described in more detail later)

The application configuration is centralized to the node, the app is actively fetched at startup, and a watcher is registered on the node, and the app is notified every time the configuration update is applied.

• Name Space Service

Distributed naming service, after creating a node, the path of the node is globally unique and can be used as a global name.

• Distributed Notification/coordination

Different systems listen to the same node, and once an update is available, the other system is notified.

• Distributed locks

Zookeeper can guarantee strong data consistency, and users can trust that the data of each node in the cluster is the same at any time. One user creates a node as a lock, another user detects the node, if it exists, it is locked on behalf of another user, and if it does not exist, it can create a node that represents a lock.

• Cluster Management

Each machine that joins the cluster creates a node that writes its own state. Users who monitor the parent node are notified and processed accordingly. Delete the node when you leave, and users who monitor the parent node will also receive notifications.

The application of zookeeper in our business logic

Our company to do Aurora push, push business platform has a large number of logical servers, grouped by business type. The operation of the logical service relies on configuration, and the configuration is adjusted online, requiring a centralized configuration item Management Center. Zookeeper's release and subscription features, as well as the mechanism for sending update notifications, are a good fit for our needs. The disaster-tolerant nature of the zookeeper also frees us from the extensive management work involved.

Below I mainly share with you the application of zookeeper in our internal service.

A. Our logical servers contain two types of configurations.

One for the ACL (Access control list), after the user's message is consumed, follow the conditions in the list to move to the next logical server. The other is simply an extrapolation of the algorithm logic, called AGL (Access algorithm list), but some of these judgment conditions change frequently. These two types of configuration are collected in the Configuration Management Center (i.e., zookeeper).

The logical diagram is as follows,

The user edits the policy configuration information (XML format) and loads the client into the zookeeper. Zookeeper immediately notifies the logical server under it (BLX) that the logical server downloads the latest configuration policy and applies the new policy. The new strategy is likely to change the flow of data for a user within a certain range of IDs, either over the original logical server, or point to the newly added logical server.

B. Data Model design

The same type of logical service creates a node on zookeeper and shares the same configuration information.
The node below is a policy configuration item, divided into ACL and AGL classes, such as: (Take proxy logical service as an example)

ACL1, Acl2, Acl3, AGL1, AGL2, respectively, have policy configuration information. Changes will notify the logical server listening to the proxy node, proxy logical server download the latest policy, and apply the policy. Joining and exiting the new node will also notify the proxy logical server.

C. Business process processes such as

1. The logical service listens to its own type node (this example, the former Diagram proxy node)
2. Edit new policy, load policy to zookeeper (policy saved in PROXY/ACLS/ACL[1..N], or PROXY/AGLS/AGL1[1..N])
3. Zookeeper notify each logical node
4. Each logical node downloads a new policy locally and applies a new policy

Original Address http://blog.jpush.cn/index.php/push_zookeeper_study_usage/

The study and application of "reprint" Zookeeper