Centralized operation and maintenance management--puppet Management Road

The characteristics of high scalability and fault tolerance in large data age put forward higher requirements for operational dimension. System Management is no longer tired of installing the operating system, configuring and optimizing the parameters of the system, patching, installing software, configuring software, adding a service. In order to improve efficiency, avoid duplication of work, reduce errors, accumulate knowledge, system administrators have begun to do some local automation work. But these are far from enough, in order to meet operational requirements, the need for more thorough application of automated operational tools.

This article will explain how to use the Configuration Management Automation tool puppet to complete system installation, monitoring and alarm work, anatomy puppet to the system administrator's convenience, but also introduces the puppet structure and working principle.

From system installation to automated deployment software, configuration, rollback, to server availability, performance, security maintenance, operational management personnel need to be fully mastered, in order to effectively complete the work, familiar with several excellent open source software is essential. As shown in table 1.

Table 1 Classification of common tools for operation and maintenance

For me, the most important thing in the Toolbox is Kickstart, Puppet, Zabbix and cacti.

difficulties in operation and maintenance

Operation and maintenance work flow

Common operation and maintenance procedures include: installation system → optimization system and configuration → installation software → configuration software → add monitoring → check. Follow-up may also be added services → configuration changes → patching fix vulnerabilities. is not feel very cumbersome. Especially when you're in charge of a lot of equipment and can't do it on your own, you need some tools to help.

The uncertainties in operation and maintenance work are even more disturbing. Changing applications on 10 machines is a simple thing to do, but it can become very complicated if you go up to thousands of units. Repetitive labor can also make people feel tired and boring, over time may also produce tired of working emotions. Using Puppet can solve these problems.

Automate yourself

In order to improve efficiency, reduce the chance of error. Many companies are gradually using automation to achieve these tasks. Some companies choose to develop a set of tools themselves, because they can be customized according to requirements, but this is really necessary. Let's take a look at the disadvantages of doing so.

1. Creating a wheel from scratch: The challenge and complexity of building a scripting job.

2. The maintainability of the program is not guaranteed (language).

3. Support different platforms.

4. After the system reload consideration.

It takes a long time to co-ordinate and plan the whole system, and with the flow of people, the level of skill varies, and new problems are brought. and a stand-alone system is unlikely to support just one platform--cross-platform development means more uncertainty.

Automated Configuration Tool Comparisons

Table 2 Compares two of the most commonly used automated operational tools puppet and Cfengine.

Table 2 Comparison of puppet and Cfengine functions

But what I really want to say is: The above comparison does not have much meaning, the tool is how you use, how to use the best, play its advantages, and your business perfect combination. We don't need to be busy selecting tools, but we should study it in depth.

Analysis of puppet

Before using any software we need to understand its working principle, otherwise it will bring a lot of inconvenience to the subsequent use. Puppet uses a very simple C/s architecture, all data interaction through SSL, to ensure security. Its workflow is shown in Figure 1.

Figure 1 Puppet Workflow

1. The client puppetd initiates an authentication request to master or uses a signed certificate.

2. Master tells the client that you are legal.

3. The client puppetd call Facter,facter detects some of the host's variables, such as host name, memory size, IP address, and so on. PUPPETD sends this information over an SSL connection to the server side.

4. Server-side Puppet master detects the host name of the client and then finds the corresponding node configuration for manifest and resolves that part. Facter sent information can be used as a variable processing, node involved in the code to resolve, other not involved in the code does not resolve. Parsing is divided into several stages, the first is the grammar check, if the syntax errors on the error; If the syntax is correct, continue to parse, the results of the analysis generated an intermediate "pseudo code" (Catelog), and then the pseudocode to the client.

5. The client receives "pseudo code" and executes it.

6. The client determines whether a file is in execution and, if so, initiates a request to the fileserver.

7. The client judge has no configuration, if configured, the execution results are sent to the server.

8. The server side writes the client's execution result to the log and sends it to the reporting system.

when the server is over thousand units

As your server becomes more and more, you may find that puppet execution efficiency is starting to drop and the server is unable to meet your needs. Here are a few scenarios for solving this type of problem.

Loadblancer

This is through a very simple expansion master scheme to enhance Master's ability to compute "pseudocode." Typically, this architecture supports about 1000 servers. Of course, it also depends on whether your system is "complex" enough.

Fig. 2 Loadblancer Scheme

There are two common implementations of this architecture: Apache+passenger, and Nginx+mongrel. This article will take the latter as an example to briefly describe its working methods.

1. Puppet Master runs multiple processes:

Puppet Master+mongrel,port 18140

Puppet Master+mongrel,port 18141

Puppet Master+mongrel,port 18142

Puppet Master+mongrel,port 18143

2. Nginx the load balance of puppet master through a upstream approach. Nginx Listener Port 8140 will be in addition to the file issued outside the request, the agent forward to the above 4 puppet master instance, Nginx will authenticate the client certificate, but need to configure the CA issued certificates to allow requests, we can also configure 8141 to provide certificate issuance.

3. If the use of Fileserver,nginx can also be directly processed.

Puppet Certification Load Balancing

Only multiple master is sufficient. A machine still has a risk, so we need to be fault tolerant, the master distributed on different machines, and CA certification is also a very important part of the following framework we can do a hot standby. As shown in Figure 3.

This architecture can also be extended. Again, we review the puppet working principle; Puppet client and Nginx are HTTPS connections, and Nginx are using HTTP connections with each mongrel. Validation of client certificates is the responsibility of Nginx, and Nginx only needs to have the public key of the CA to do the validation work. The advantage of this is that multiple managers do not need to sync the client's certificate settings, only need to have the public key of the CA, public key replication can be used. But there is one drawback: it's not easy to delete a client certificate. However, a master manager can be used in a way that other management machines synchronize certificates from this manager in real time.

Fig. 3 Puppet certification load Balancing scheme

Puppet Management machine cluster ideas are as follows:

1. Synchronize the CA configuration to each machine, including the public private key;

2. Use CA to issue a certificate to each management machine;

3. Each management machine is equipped with a loadbalancer mode, 8140 provides configuration management, 8141 provides certificate issuance;

4. Keeplived can be used between managers to achieve high availability and failover, including Ha, the architecture can be arbitrarily extended;

5. Each management machine configuration cent production and development two kinds, simply publishes to the management machine through git;

6. The test only modifies the development part, assigns it in the individual client, succeeds then pushes to the production;

7. Configure a master CA management machine To resolve the problem of removing authentication.

Rational Planning

All of the ex-post rescue plan is not as good as before the use of reasonable planning, you need to be very clear about the current business status, operational dimensions of the status quo. Understand what you need to solve, then break it down, and then step through it. It is recommended to use Git management puppet, standardize hostname, adopt DNS Management, Fileserver Independent, put the infrequently changed in the fileserver, often change in the template, and communicate the custom OS.

Maybe a lot of people don't quite understand why they're customizing the OS, its biggest advantage is that the system can be initialized when the installation to help you do some puppet needed software package, through the purchase of equipment, the SN, the WebUI system to register the machine information, the machine can be completed after the start of all configuration. If your WebUI do a better job, you can call the Monitoring System API to complete the monitoring. This is not perfect.

Concluding remarks

I believe that after reading this article, not only the overall understanding of puppet, but also more familiar with the focus of automated operation and maintenance work. You may start thinking about using Puppet to solve a lot of repetitive work in your own operation, or to solve the architectural problems you are facing.

I want to say to many system administrators who wish to learn Puppet or are using Puppet, the principle of work is very important, many people just do not understand the principle of work, so in the process of using a problem on the run. Reader friends must rely on more brain thinking to solve the problem.

author Yulaw, one of the founders of Linuxtone.org, Sinaedge platform, chief of operation and maintenance. Responsible for Sina Weibo, Sina video, watch, micro-disk, music and other business CDN operation. has written "puppet centralized management."