"Organizing and Learning Hadoop": The second foundation of Hadoop Learning-distributed

Distributed system is the application layer of the system, the data layer or other parts of the architecture component fabric (physical and logical can be) (usually mesh). Distributed systems are often designed to enhance the scalability, stability, and execution efficiency of the system. For example, online games are usually distributed systems, where the so-called "district" is the distribution system lining routines. The distributed database can also be called Distributed system, the data Persistence layer is distributed (data exist in different databases, can interact, there is a set of IMIS to maintain data integrity and accuracy). So distributed systems are more accurately a system architecture concept, not a technology.

In a distributed system, a set of independent computers presented to the user is a unified whole, as if it were a system. The system has a variety of common physical and logical resources, can dynamically assign tasks, decentralized physical and logical resources through the computer network to achieve information exchange. There is a distributed operating system that manages computer resources in a global manner. Typically, a distributed system has only one model or paradigm for the user. There is a layer of software middleware (middleware) on top of the operating system that implements this model. An example of a well-known distributed system is the World Wide Web, in which everything looks like a document (Web page) in the World Wide.

In computer networks, this uniformity, model, and software are not present. The user sees the actual machine, and the computer network does not make the machines look uniform. If these machines have different hardware or different operating systems, these differences are completely visible to the user. If a user wants to run a program on a remote machine, he must log on to the remote machine and run the program on that machine.

The common point of Distributed system and computer network system is that most distributed system is built on computer network, so the distributed system and computer network are basically the same in the physical structure.

The difference between them is that the design idea of the distributed operating system is different from that of the network operating system, which decides that they are different in structure, working style and function. Network operating system requires network users in the use of network resources must first understand network resources, network users must know the network of the functions and configuration of each computer, software resources, network file structure, etc., in the network if the user to read a shared file, The user must know which directory of the computer the file is placed in, and the distributed operating system manages the system resources in a global manner, which can dispatch the network resources to the user arbitrarily, and the scheduling process is "transparent". When a user submits a job, the distributed operating system can select the most appropriate processor in the system as needed, submit the user's job to the handler, and pass the result to the user after the processor finishes the job. In this process, the user does not realize that there are multiple processors, and the system is like a processor.

Compared with the centralized system, another potential advantage of the distributed system lies in its high reliability. By dispersing the workload across a multitude of machines, a single chip fault can only shut down one machine, and no other machine will be affected. Ideally, at some point, if 5% of the computers fail, the system will still be able to continue to work, only losing 5% of its performance. For critical applications, such as nuclear reactors or aircraft control systems, the use of distributed systems to achieve the main consideration is that it can achieve high reliability

The business object is deployed on the client and server side in the same situation, depending on the configuration file to choose whether to connect to the server or connect to the local database, this problem let me racked my brains, I always imagine the client and server-side communication is the lowest-end socket. It took two nights to study the csla.net framework about the data portal code, only to find the crux of the problem: client-server communication can not use the lowest-end socket, but to use high-end webservice,. NET remoting or define a protocol, etc. As long as they support the client directly according to the server-side service URL, class name, method name and method parameter four information can call the server corresponding class and method on the line.

Note: This article expresses the thought and the csla.net to have the very big difference, does not see this article thought is csla.net the design idea, also does not think this article wrong explanation csla.net, this is not an introduction csla.net the article, but pure thought they are identical.

• Deployment of Distributed Systems

　Usually we say three-layer architecture, I think it is a generalized model, more layers of design can merge adjacent layers of the way eventually back to three layers of this broad concept, I mean: these are just concepts, forget these concepts to practical analysis design will be closer to these concepts.

Next I want to change the three layer to a simpler point, two layers, the data access layer merged into the business layer, collectively referred to as the business layer, because we are faced with the problem is not a layered problem, but the distribution system in how each layer should be deployed. The Csla.net book also says that the business layer and the data access layer can be placed on the same machine to improve performance and fault tolerance. Therefore, their merging does not affect the deployment of distributed systems.

However, it is explained that the database system (the data storage and management layer in csla.net) does not take into account the three layer, which is not included in the data access layer, if it is counted in, then it is under the data access layer alone.

In conclusion, the stratification in the Distributed system deployment is actually three layers: the interface layer, the business layer (which includes the business layer of the data Access layer), and the data storage layer.

　　Below is an example of a possible deployment scenario where a shaded box indicates a machine with a dashed border indicating that it is optional for use, and a dashed line indicates that a separate out server is drawn from here. In the B/S application, the Web browser is the client, and all the others are servers. In the C/S application, the interface layer at the top level + the business layer is the client, the other is the server .

Deployment of non-distributed systems

650) this.width=650; "height=" width= "314" src= "http://images.cnblogs.com/cnblogs_com/pains/201106/" 201106060524246487.jpg "alt=" standalone version "border=" 0 "title=" standalone "style=" margin:0px;padding:0px;border:0px;background-image : none; "/>

Stand-alone version

650) this.width=650; "height=" 394 "width=" 167 "src=" http://images.cnblogs.com/cnblogs_com/pains/201106/ 201106060524265700.jpg "alt=" Layered 2 "border=" 0 "title=" Layered 2 "style=" Margin:0px;padding:0px;border:0px;background-image : none; "/>

Two or three sets of machines

Deployment of Distributed Systems

650) this.width=650; "height=" 498 "width=" 545 "src=" http://images.cnblogs.com/cnblogs_com/pains/201106/ 201106060524331593.jpg "alt=" Layered 3 "border=" 0 "title=" Layered 3 "style=" Margin:0px;padding:0px;border:0px;background-image : none; "/>

A distributed web system

650) this.width=650; "height=" 404 "width=" 545 "src=" http://images.cnblogs.com/cnblogs_com/pains/201106/ 201106060524404704.jpg "alt=" Layered 4 "border=" 0 "title=" Layered 4 "style=" Margin:0px;padding:0px;border:0px;background-image : none; "/>

Distributed c/S system

There are a few points to note:
1. Business logic such as validation on the client is not trustworthy, so any deployment requires the server side to include the business layer;
2. For high scalability in development, maintenance, and deployment, each business layer in the diagram contains code that is identical;
3. Because 2nd, so I encountered the same operation of the business layer is to communicate with the business layer on other machines or access to the database is the problem.

Address data access issues at the business level

This question is a key issue, the 3rd of the above-mentioned points, in order to solve this problem we introduce the concept of data portal.

The following is an example of WebService: the interface layer access to the business object of the "check" method, skip the database query operation, access to another machine in the same business object class "Check" method.

650) this.width=650, "height=", "width=" 780 "src=" http://images.cnblogs.com/cnblogs_com/pains/201106/ 20110606052448847.jpg "alt=" request "border=" 0 "title=" request "style=" Margin:0px;padding:0px;border:0px;background-image: none; "/>

　The above is to send a request to another machine, the request does not directly call the business object class on another machine "check" method, but the business object and method parameter information to be invoked into a "binary package" as a parameter to call another machine on the common "check" method, the other machine "check" The method then unlocks the package and then goes to invoke the business object type represented in the unpacked package, and the following static diagram is the work that another machine receives after the request.

650) this.width=650; "height=" 726 "width=" 736 "src=" http://images.cnblogs.com/cnblogs_com/pains/201106/ 201106060524566085.jpg "alt=" response "border=" 0 "title=" response "style=" Margin:0px;padding:0px;border:0px;background-image: none; "/>

1. The principles have been described in the diagram, not another large paragraph of text explained,
2. In the above two diagrams, except for the "actual business object class", all belong to the structure or frame part;
3. If you use OO thinking to review the above two charts, you will be complaining about the bad design, here just to describe the work of the distributed system as simple as possible, you can use the policy mode to adapt the data portal without changing the circumstances of the various requests to respond to the situation, the use of Factory mode to achieve different requests to respond to the occasion of the switch.

 About the distribution of databases

　In order to solve the burden of the database server, we may want to store the data distribution on multiple servers, I envision the database distribution scheme, the database on each server is structurally identical, and the table of data stored on different servers, In this way, the data access layer sends the same SQL commands to all the database servers when the data is checked, and then the data access layer is consolidated after the data is received, thus reducing the workload for each server. or distribute data to different servers based on a representative field in the table (for example, province).

"Organizing and Learning Hadoop": The second foundation of Hadoop Learning-distributed