[Reprint] Chat cluster management mode

Original: http://dockone.io/article/423

The article summarizes very good, is a popular science article, but has always had a question, immutable servers how to solve frequent local small file update problem?

Docker is very hot and very red, almost to the point of no reason. Why is Docker so red? Because it's a technique that can be used to lift tables. In the industry of deployment automation workers and machine tool manufacturers, housekeeping guard CMDB, distributed script execution and so on so-called core technology, even if not become the end of the world, will become a second-rate technology. Just using Docker as a lightweight VMware is impossible to see through its essence. To understand what Docker means, you can't start with what Docker is and what you can do about it. Let us recall the development of cluster management mode, and the shortcomings of these backward models.

The era of manual management

The IP address is placed in the Excel table. Management is to rely on landing board machine, SSH connection server. Manually execute the command to do a new server deployment, a version of the server has been upgraded, and a variety of configuration refresh modification work.
The drawbacks are self-evident, there are several points:

• Lack of consistency, because it is manual operation so there are always some differences between servers
• Inefficient, the number of servers a person can manage is very limited
• The era of automation leap forward

The increase in the number of businesses quickly makes the number of machines more than the limit of manual maintenance. No matter how bad the team, as long as the business to this part, there will inevitably be a large number of automated tools to automate the implementation of scripts to support the business quickly. This time is a golden age, the operation of the real face of the times. Because there is no automated operation and maintenance technology, the business will encounter bottlenecks. The introduction of automation technology effectively embodies the revenue of the business.

This era is characterised by two key systems.

• The IP address in the local Excel table is managed by the database, called the CMDB.
• Distributed script execution platform based on SSH or agent

Inefficiency is no longer a major problem, and the main drawback becomes:

• A large number of scripts, clutter, content duplication, quality is difficult to guarantee, and eventually left a hidden trouble
• There is no definition and management of the current network expected state, all the current network status is the product of the cumulative script, resulting in server state drift, the production of Snowflake server (each machine is not the same), thus leaving the business stability hidden trouble

These shortcomings in the short-term business is not immediate harm, belongs to internal injury type. And many hidden dangers, even if exposed, will flow in the emphasis on discipline, emphasizing operational awareness and so on. Few people are going to pursue the concept of operations behind the problem. The result is that most companies are stuck at this stage. After all, operation is a sufficient area of support. Operation and maintenance of high-tech, high-availability, not necessarily with the success of the entrepreneurial company how many direct links.

Development Revolution era

Along with DevOps comes the infrastructure as code. Simply put is a bunch of development to kill to the field of operations, see these operations are actually this way to manage the status of the current network. So they brought the experience of writing code and set the current state into a model (so-called code) and submitted the expected State to version control. Just like writing code, go to manage server configuration.

Many back-end development-led small startups directly skipped the previous era, and the OPS automation system was based on puppet and chef from the start. In all fairness, the use of puppet more is a lack of historical baggage, not because of the complexity of the operation and maintenance problems. Many of the machines that are managed are not more than 10 units, but there are also teams that waste a lot of time on how to use Puppet/chef. On the contrary, many large companies because of the heavy historical burden, and the large traditional operations team, this development revolution road is not going to get through.

This approach is mainly to solve the problem of scripting management, and because the direct definition of the current state of the network, the consistency between the server is much better. But the shiny model behind it is essentially a bunch of scripts to drive. The drawbacks of the last era were only packaged and improved, and there was no way to eradicate them.

Apply the expected state to the current network depends on or run the script. And unlike before, now more is to run others write cookbook, quality is mixed.

Although the expected status of the current network is defined, the upgrade operations that are different starting points (such as from A=>c, b=>c) may be completely different. It's really hard to write an exhaustive upgrade script.

What else is the problem?

Consistency and stability are the biggest problems. After the server is powered on, the system is not re-installed all year round. Countless people have run through scripts, executed commands, and positioned problems. The actual state of the server is no way to precisely control it. Infrastructure as code is an improvement, but it still does not eradicate the problem. Running a script on a server every time is a gamble, because no two servers are exactly the same. Testing a workable script locally may not cause problems on another stage. This is not an emphasis on the code can not RM *, and the RM path/* to solve the problem.

Version management has never been in fact. People who have done development may also use GIT/SVN as a baseline for deployment, and the basic version will be submitted to the repository. More of the first-line OPS is the rsync model. Rsync means to install a new server, you need to find a "with the most image" of the server. Then copy the files to the new server, modify the configuration, and start the finished. There was something wrong with Ctrip, and my personal guess is that it has to do with version management clutter.

Failure to replace is very difficult. First, do not say fault replacement, is the fault machine culling is a headache thing. Like zookeeper. Each client is hard-coded with three IP addresses. Once one of the IPs is hung out. Zookeepr can remain normal according to the high-availability protocol, but in the long run the hanging IP will be removed from the various users. And that's changed. Once the high-availability of the business is not good enough, need operations to engage in a number of alarm after the replacement of the fault machine, it is a variety of scripts to toss the rhythm of various configuration files.

How Docker lifted the table.

Two point of view, after entering the age of Docker

• The CMDB is no longer critical. The CMDB, along with IP and server resources, became a concern for the bottom blue-collar workers. Background development and business operations on the upper level no longer need to be able to manage the configuration with the IP-centric CMDB.
• The distributed script execution platform is relegated to the second tier from the core operating system. Very simple, the server no longer needs to change, the general new server, the release of the new version is no longer dependent on the script on an existing server to perform the modification state. Instead, create a new container.

The essence of

Docker is a true version management tool. Prior to Docker version management was a patchwork of solutions. What is a version, the server is made up of three parts: version, configuration, data. The so-called version is the operating system, as well as the operating system configuration. Various third-party packages, developed to executable files, and a subset of configuration files. These collections are a version, in fact, a complete executable environment. In addition to the general is a database, there are two parts of the content, part of the administrator can modify the configuration from the page, part of the business data. In the puppet era version, is a declaration document. This declaration file execution, you need to install an operating system from an ISO, and then use Apt-get/yum to install a bunch of system packages from a mirror source, and then install a bunch of Python/ruby language-level packages with Pip/bundle, and finally the git/to develop for you svn/an unknown tar.gz. Do you think these things are in the same version every time they're assembled? Not really. When I wanted to kill GitHub on a wall, I didn't know how many people couldn't make a release. The image that Docker packs together with the system is actually the best description of the version.

no longer needs to modify the container of the current network after using Docker. A container If you need to upgrade, then kill it, and then release the pre-made new image into a new container replace it. Distributed script execution becomes a distributed container replacement. Of course, this standardized operation, with Mesos Marathon has been perfectly solved.

After using Docker, it can no longer be managed based on IP. Instead of assigning an IP to each container, the static model of the IP representation cannot keep up with the times. Based on IP management, it means that you will be based on SSH login to this IP to manage. This kind of thought is backward from the bone. Process, process group, module, set these are the granularity of management. It is no longer important that the process is in which container on which IP to run. A graph can illustrate this problem:

Do you want to fill in the IP after this expansion button is finished? No! You just have to tell marathon that I want 32 process instances. It will look for these resources to run these 32 instances. The business ultimately needs 32 processes, not 32 IPs. IP is just the resource you need to run the process. The actual run time process may be on an IP start 32 ports, it may be randomly assigned 5 IP, each ran some ports. Of course, these allocations can be expressed in a "constrained" manner. Instead of letting you get 32 IPs, run a script to deploy those processes on those IPs.

The Missing Piece

This is the last piece of puzzle game. Docker as a version of the tool is absolutely qualified. Marathon is also a reliable way to host all processes in Docker. But it's not complete:

• The Docker image will not work after it is released as a version, because there are at least a few services that are accessible to each of the applications. These hard-coded IP addresses in the mirror cannot be executed in a different environment. Any configuration in a version can be hard-coded, that is, the IP address and port is not hard-coded.
• Expanding the scale can easily create and destroy containers, but what about other containers that reference the container's servers?
• Release, fault replacement is the same problem

The solution can look at these two graphs:

The scheme is actually very simple. The APP1 = APP2 Network access relationship, changed to App1 =local=> haproxy =network=> haproxy =local=> app2. By deploying Haproxy "all ports" locally on the container, that is, by using Haproxy to do the wiring between processes, instead of each process being responsible for connecting to other processes on the network itself.

Think of the previous hard-coded 10.0.0.1:3306 in the config file as a database. Hard coding is wrong and it's about spanking. So we changed the hard-coded IP address to 127.0.0.1:10010. This time we no longer hardcoded any IP, we only hardcoded a special port number. Each process has a bunch of special local port numbers to access the upstream and downstream services that it needs. The process behind this port number is in which IP, which port, which container is executed. As the user does not need to modify any code (such as compatible with what ZOOKEEPER/ETCD God horse), do not care. Even this port is followed by multiple remote IPs that make up a client-based high availability. The agent can even do something wrong to change a backend and try again.

With this artifact, the capacity to expand, release changes, failure to replace is easy. Containers are freely added and deleted. Once the network structure changes, refresh the Haproxy configuration for each location. A variety of grayscale, a variety of 0 downtime replacement solutions can be started.

Name Service and network

There are a lot of similar schemes. The bottom-up scenario is sdn/ip drift, and the bonding of the network. The feature of this scheme is to maintain the IP address as the most traditional name service, in an attempt to perpetuate its life.

The upper-level scenario is DNS. A few more plans are ZooKeeper.

A variety of solutions are competing for how services register themselves, and how to discover this point. The pros and cons of various schemes can be read on their own:

• Smartstack:service Discovery in the Cloud
• Dockercon video:building A SMARTER application STACK

Btw,airbnb put the package into production in 13.

The most interesting thing is to compare this haproxy scheme with the SDN-based IP drift scheme. The haproxy is to do the wiring of the application layer process for the network, by introducing Haproxy to make the line more flexible. And the SDN scheme is to say that you are now the business process between the static link between the IP, this connection is not flexible enough, the router helps you the whole. One IP hangs up, you can drift the IP to another machine to continue to use. In fact, in a scene to achieve the re-linking of two processes, breaking through the static exchange of two IP restrictions, to the IP-based deployment scheme to renew life.

The technology at the bottom of the two is interlinked. The so-called IP drift is finally relying on modern-day CPU, and software routing technology. The last play is user-state forwarding, DPDK god horse. So haproxy slow, forwarding efficiency has problems God horse, in the long run will not be a problem. Using software to line up is a trend. Even the routers are starting to play, and even the hardware vendors are starting to sell the software.

The Final Battle

Cluster management becomes process management purely, IP is no longer important, and status is no longer important. The CMDB will become more and more marginalized.

The release change is no longer about modifying the server, but creating a new destruction container and updating the inter-process network connection relationship. The distributed operating system will become less used, and the board is not allowed to use it.

Remember the phrase "immutable Servers", which will eventually be recognized by history.

[Reprint] Chat cluster management mode