"Finishing Learning Hadoop" One of the basics of Hadoop Learning: Server Clustering Technology

Serverclustering refers to the pooling of many servers together for the same service,ClientIt looks like there's only one server. Clusters can be used by multiple computers toParallel computingThis results in a high computational speed and can be done with multiple computersBackup, which makes any one machine bad for the wholeSystemwill still work.

Cluster (cluster) technology is a relatively new technology, through clustering technology, can be paid at a lower cost in the performance, reliability, flexibility of relatively high returns, and its task scheduling is the core technology in the cluster system. A cluster is a set of independent computers interconnected by high-speed networks, which form a group and are managed in a single system mode. When a customer interacts with a cluster, the cluster is like a separate server. Cluster configuration is used to improve availability and scalability.

Objective:

1 Lift Performance

Some computationally intensive applications, such as weather forecasts, nuclear test simulations, and so on, require computers to have a strong computational processing power, and the existing technology, even if the ordinary mainframe computing is very difficult to perform. At this time, the general use of computer clustering technology, centralized dozens of or even hundreds of computer computing capacity to meet the requirements. Improving processing performance has been one of the important goals of cluster technology research.

2 Reduce costs

Typically a better cluster configuration with more than $100000 in hardware and software overhead. But it's quite cheap compared to a dedicated supercomputer worth millions of dollars. Under the same performance condition, it is more cost-effective to use computer cluster than the mainframe computers with equivalent computing ability.

3 Improve Scalability

Users who want to expand their system capabilities will have to buy a higher-performing server to get the extra CPU and storage they need. If you use cluster technology, you only need to join the new server into the cluster, for the customer, the service, regardless of continuity or performance is almost unchanged, as if the system has been unknowingly completed the upgrade.

4 Enhanced Reliability

Cluster technology allows the system to continue to work in the event of a failure, minimizing system outage time. The cluster system improves the reliability of the system, and also reduces the failure loss greatly.

Advantage:

First, the cluster system can solve all the server hardware failure, when a server has any failure, such as: Hard disk, memory, CPU, motherboard, I/O board and power failure, the application running on this server will switch to other servers.

Second, the cluster system can solve the problem of software system, we know that in computer systems, users are using applications and data, and the application system running on the operating system, the operating system is running on the server. Thus, as long as the application system, operating system, server, any one of the three failures, the system actually stopped to provide services to the client, such as our common software crashes, is one of the cases, although the server hardware intact, but the server still can not provide services to the client. The biggest advantage of clustering is that the monitoring of the failed server is based on the application, that is, as long as the server's application is stopped, other related servers will take over the application, regardless of the reason for the application to stop running.

Third, the cluster system can solve the human error caused by the application system to stop working, for example, when the administrator on a server is improperly operating caused the server downtime, so the application running on this server has stopped running. Because the cluster is monitoring the application, other related servers will take over the application.

in theCluster system, all computers have a common name, and services running on any system within the cluster can be used by all network clients. The cluster must be able to coordinate the errors and failures of each detached component and transparently add components to the cluster. The user's public data is placed in a shared disk cabinet,Applicationis installed on all servers, which means that applications running on the cluster need to be installed on all servers. WhenCluster systemduring normal operation, the app runs on only one server, and only this server can manipulate the data area on the shared enclosure, and the otherServer monitoringthis server, as long as the application on this server stops running (eitherHardwaredamage, operating systemCrash, application softwareFaultor an application that is caused by a human error operation to stop running), the other server takes over the application that the server is running and takes over the corresponding data area on the shared enclosure.

Server Cluster Technology the definition

The rapid development of network technology has brought great challenge to network bandwidth and server performance, and more and more bottleneck of Internet-based application system appears on server side, which puts forward higher requirements to server. A cluster is a set of servers that work together to provide a more stable, efficient, and scalable service platform than a single service. In the external view, the cluster is a separate service entity, but in fact, there are two or more than two service entities within the cluster to coordinate and cooperate to complete a series of complex work.

Clusters are typically made up of two or more servers, each of which is called a cluster node, and the cluster nodes can communicate with each other. There are two ways of communication, one is based on RS232 line heartbeat monitoring, the other is a separate network card

To run a heartbeat. Therefore, the cluster has the node service status monitoring function, but also must have the Service Entity expansion function, can flexibly increase and eliminate a service entity.

In a cluster, the same service can be provided by multiple service entities. Therefore, when one node fails, the other node of the cluster can automatically take over the resources of the failed node, thus guaranteeing the service to run persistently and uninterrupted. Thus the cluster has a fault

Automatic transfer function.

A clustered system must have a shared data store because the services provided by the cluster are consistent, and when any one of the cluster nodes is running an application, the applied data is centrally stored in the node's shared space, while the operating system on each node runs only the service of the application and stores the application files.

To sum up, the construction of a cluster system requires at least two servers, but also need to have serial cable, cluster software and shared storage devices.

Server Cluster Technology features and functions of

High Availability and scalability

1. Highly adaptable

For some real-time applications, it is necessary to keep the service running 24 hours a day, and because of the software, hardware, network and human factors, a single service operating environment is difficult to meet this requirement, it is a good choice to build a server cluster system at this time. One of the biggest advantages of building a cluster is the high availability of the cluster, in the event of a service failure, the server cluster system can automatically switch the service from the failed node to another standby node to provide uninterrupted service and ensure the continuous operation of the business.

2. Extensibility

With the increase of business volume, the existing Cluster service entity can not, meet the requirements of the cluster to dynamically join one or more service nodes, so as to meet the needs of the application, enhance the overall performance of the cluster, which is the scalability of the cluster.

Load Balancing and error recovery

1. Load Balancing

The most important feature of server cluster system is that it can flexibly and effectively share the load of the system, and assign the client's access to the following nodes through the cluster's custom load sharing policy. For example, you can define a polling allocation policy, distribute requests evenly across service nodes, and define a minimum load allocation policy, and when a request is made, the server cluster system determines which server node is relatively idle and distributes the request to this node.

2. Error recovery

When a task is not completed at one node, and for some reason the execution fails, another service node should be able to perform this task, which is the error recovery function provided by the server cluster, which ensures that each execution task is completed effectively with the wrong redirection.

Heartbeat Monitoring and Drift IP

1. Heartbeat Monitoring

In order to achieve load balancing, provide highly available services, and perform error recovery, the server cluster system provides heartbeat monitoring techniques. Heartbeat monitoring is achieved through the heartbeat line, you can do a heartbeat line of the device has an RS 232 serial line, you can also use a separate network card to run the heartbeat, can also be a shared disk array. The number of heartbeat lines should be reduced by 1 for the number of server cluster nodes. Note that if the heartbeat through the network card, each node needs two network card, wherein, a private network as a direct connection to each other and its corresponding network card, used to monitor the other heartbeat; another NIC connected to the public network to provide services, At the same time the heartbeat network card and service network card IP address try not to be in a network segment. The efficiency of heartbeat monitoring directly affects the length of failover time, and the server cluster system keeps the internal communication of the nodes effectively through the heartbeat technology.

2. Drift IP Address

In a server cluster system, there is a drift IP address in addition to the real IP address of each service node itself. Why is Drift IP? Because this IP address is not fixed, for example, in a two-node dual-machine hot standby, the normal state, the qualified drift IP is located on the primary node, when the primary node fails, the drift IP address automatically switch to the standby node. Therefore, in order to ensure the uninterrupted nature of the server. In the cluster system, the external service IP must be the drift IP address, although the IP of the node itself can also provide services, but when the node fails, the service switches to another node, the service IP is still the IP address of the failed node, then the service is interrupted.

Server Cluster Technology the classification

Highly available clusters

1. Highly Available concepts

The English full name of the highly available cluster is high availability, referred to as HA cluster. The meaning of high availability is to be used to the maximum. As can be seen from the name of the cluster, the function of such a cluster is to ensure that the user's application provides service in a persistent and uninterrupted manner.

When an application fails, or the system hardware or network fails, the application can automatically and quickly switch from one node to another to ensure that the application continues to provide uninterrupted service to the outside, which is the function of the highly available cluster implementation.

2. Common Ha Cluster

Often said two-machine hot-standby, dual-machine interoperability is the category of high-availability clusters, such clusters are generally composed of two or more than two nodes. A typical dual-machine hot standby structure is shown in 15-1.

650) this.width=650; classification of "alt=" Server Cluster Technology "src=" Http://server.zzidc.com/uploads/allimg/131121/1-131121143F4L6.png " Title= "Classification of server cluster Technology" style= "padding:0px;margin:0px;border:none;width:306px;height:165px;"/>

Dual-Machine hot standby is the simplest application mode, that is, often said Active/standby way. It uses two servers, one as the primary server, runs the application for external service, and the other as a standby machine, installs the same application as the master server, but does not start the service and is in standby state. Between the host and the standby through the heartbeat technology check each other, monitoring resources can be network, operating system, can also be a service, users can according to their own needs, select the resources to monitor. When a backup monitor to a host of a resource failure, according to pre-set policies, the IP first switched over, and then the application services are also taken over, and then by the standby machine to provide services. Switching process time is very short, the user can not feel the program is not a problem, but also to switch, so as to ensure the application of durable, uninterrupted service.

Dual-machine Mutual preparation is on the basis of dual-machine hot standby, two independent applications on two machines simultaneously running, mutual preparation, that is, two servers are host and standby machine. When any one application fails, another server can take over the application of the faulty machine in a short period of time, thus guaranteeing the continuous and uninterrupted operation of the service. The advantage of the dual-machine is that it saves equipment resources, two applications require at least 4 servers for dual-machine hot-standby, while the dual-machine interoperability requires only two servers to complete the high-availability cluster function. But the dual-machine interoperability also has its own shortcomings: After a node failover, the other node on the same time running the service of two applications, there may be excessive load situation. Multi-machine Mutual preparation is a technology upgrade of dual-machine hot standby, through multiple units and the formation of a cluster, you can set up a flexible takeover strategy between multiple machines.

Attention

High-availability clusters do not guarantee the security of the application data, he only solves the external provision of persistent service, the software, hardware, network and human factors caused by the failure of the application to minimize the impact of the program.

3. Highly Available cluster software

High-availability clusters are typically implemented with highly available software. The high-availability software commonly used under Linux has hearbeatha,red Hat's RHCs, commercial software rose,keepalived and so on.

Load Balancing cluster

The load Balancing cluster is also comprised of two or more servers. It is divided into two parts of front-end load dispatch and backend service. The load dispatch partial load assigns the client's request to the backend service node according to different policies, and the backend node is the part that really provides the nutrition program service. Unlike Ha cluster, in a load-balanced cluster, all back-end nodes are active, and they both provide services externally and share the workload of the system.

A load Balancing cluster can distribute a high-load application to multiple nodes, which is suitable for busy and heavy-duty applications. But it also has a shortage of places; When a node fails, the front-end dispatch system does not know that the node is already unable to provide the service, and will still dispatch the client's request to the failed node, so that access will fail. In order to solve this problem, the load dispatch system usually introduces the node monitoring system.

The node monitoring system is located on the front load dispatch machine and is responsible for monitoring the following service nodes. When a node fails, the node monitoring system will automatically remove the failed node from the cluster, and when this node returns to normal, the node monitoring system will automatically join the cluster, all of which is completely transparent to the user. Figure 15-2 shows the basic structure of a load-balanced cluster.

650) this.width=650; classification of "alt=" Server Cluster Technology "src=" Http://server.zzidc.com/uploads/allimg/131121/1-131121143Q14O.png " Title= "Classification of server cluster Technology" style= "padding:0px;margin:0px;border:none;width:352px;height:205px;"/>

A load Balancing cluster can be implemented either by software or by a hardware device.

Scientific Computing Clusters

High-performance computing clusters, referred to as HPC clusters. Such clusters are dedicated to providing powerful computing power that a single computer cannot provide, including numerical computation and data processing, and tends to pursue comprehensive performance. HPG is similar to supercomputing, but different, and computing speed is the first goal of Supercomputing pursuit. The fastest speed, maximum storage, the largest volume, and the most expensive price represent the features of supercomputing. With the increase of the demand for computing speed, supercomputing is also applied in various fields, and the pursuit of the single computational speed index for Supercomputing is a comprehensive index of high performance.

HPC is widely used in applications such as scientific research, genetic test comparisons, data mining applications, oil and gas exploration, and image presentation.

See also: http://server.zzidc.com/fwqjs/322.html

"Finishing Learning Hadoop" One of the basics of Hadoop Learning: Server Clustering Technology