Analysis of coordinated design of distributed core technology

Fourinone Introduction

Fourinone (Chinese name "Sibuxiang") is a Tetraena distributed computing framework, before writing this framework, I have a long time to think about distributed computing, but also read the other open source framework written by foreigners. When we learn about complex Hadoop as a subject, seems to have forgotten what we wanted to solve the problem: we just wanted to write a program to use a few or more machines together to compute, to use more CPU and memory, to solve our large number of problems and computational complexity. Of course, this process should take into account the distributed coordination and fault handling. If only to achieve this simple purpose, why is everything so complicated. I think I can write a simpler thing, it doesn't need to be overly designed, it just needs to look cooler, smaller and more powerful. So I put my own understanding of the distributed into this framework, taking into account the similarity of the underlying implementation technology, I will hadoop,zookeeper,mq, distributed caching the four major distributed computing functions in a framework for complex distributed computing applications have been a large number of simplification and induction.

FOURINONE2.0 provides a 4-in-1 distributed framework and Easy-to-use programming API to achieve a unified use of multiple computers ' CPUs, memory, and hard disks, thus acquiring powerful computing power to solve complex problems. Due to the physical independence of multiple computers, the Fourinone framework also provides complete distributed collaboration and locking as well as simplifying MQ functionality for multiple-machine collaboration and communication.

The Fourinone framework provides a range of parallel computing models (peasant/Contractor/Job description/Manual warehouse) for the use of multiple-machine multi-core CPUs in computing capacity;

Provides a full distributed cache and small cache for leveraging multiple computer memory capabilities;

To provide the ability to operate a remote file (access, parallel read/write, split, exclusive, copy, parse, transaction, etc.) to operate as a local file for use with multiple-machine hard disk storage;

Due to the physical independence of multiple computers, the Fourinone framework also provides complete distributed collaboration and locking as well as simplifying MQ functionality for multiple-machine collaboration and communication.

Fourinone Java Development, 2.0 version of the overall size of 150k, on a jar and a configuration file, without any dependencies, very convenient for embedded development use.

Zookeeper Introduction

Zookeeper is a collaborative implementation of the Hadoop ecosystem, providing coordination services including distributed locks, uniform naming, and so on. Chubby is a collaborative implementation in Google's distributed system, similar to zookeeper. Zookeeper is actually an open-source implementation of Google's Chubby. The Paxos algorithm was proposed by Leslie Lambert in 1989 and is widely used to handle message-passing consistency in distributed systems.

The Zookeeper Configuration Center implementation is more like a file system in which all files in the file system form a tree structure. Zookeeper maintains such a tree hierarchy, where the nodes in the tree are called Znode, and each Znode store has a size limit of less than 1m. Zookeeper offers several types of Znode: temporary znode, persistent znode, sequential node, etc., for different conformance requirements. When Znode changes, the watch mechanism allows the client to be notified. You can set observations for the "operations" of the zookeeper service, and other actions of the service can trigger observations. The operations of the Zookeeper service include some additions to the Znode add modification. Zookeeper uses a similar Paxos algorithm to achieve leader election, which is used to solve the consistency and coordination of cluster downtime. Overall, Zookeeper provides a distributed collaborative system, including configuration maintenance, name service, distributed synchronization, group services and other functions, and the relevant operational interface to the user.

Implementation principle

The implementation of Fourinone to distributed collaboration is accomplished by establishing a node information of domain and node two-layer structure. Domain can be a taxonomy or a package, and node can be a specific attribute. Domain and node are themselves named after the requirements, such as domain name can be named "A.B.C ..." Represents a tree-like category.

A domain can have many node, each node only specify a domain, you can return it through domain all the node below it.

Domain does not need to be created separately, usually when you create node, if domain is not present automatically.

If there is no node in domain, the domain is automatically deleted.

If you delete domain, node under this domain will also be deleted.

A value can be stored under each node, and can be any object.