A small example of slow puppet operation

A small example shows how Debug puppet runs slowly.

A small example shows how Debug puppet runs slowly.

The agent of a machine is slow. First, let's look at the overall performance of the puppet Server:

Puppet server uses the passenger + nginx structure, which has stable performance and connects about 1200 machines. Passenger is similar to PHP-FPM and can set parameters such as the number of processes started, used to adjust the server performance.

For example, the passenger_max_pool_size parameter takes two factors into account for setting this value. The number of available memory of the server itself (the number of memory occupied by each process can be viewed through passenger-memory-Stats ), whether queue accumulation is processed (view via passenger-status)

The previous machine was a 4-core CPU, and the user CPU was more than 90% in a day. The 4-core machine load was more than 20 in a day, and the client running puppet would time out, after a migration, the performance will increase significantly.

Before migration:

650) This. width = 650; "src =" http://s3.51cto.com/wyfs02/M02/4A/BD/wKiom1QmznKwaJWtAAMHCa_Q2q0393.jpg "Title =" 1.png" alt = "wkiom1qmznkwajwtaamhca_q2q0393.jpg"/>

650) This. width = 650; "src =" http://s3.51cto.com/wyfs02/M02/4A/BF/wKioL1QmzsuC57QGAANzW1BpRZo114.jpg "Title =" 2.png" alt = "wkiol1qmzsuc57qgaanzw1bprzo114.jpg"/>

After the migration, adjust the pupept running interval to 10 minutes:

The machine is 24-core.

650) This. width = 650; "src =" http://s3.51cto.com/wyfs02/M02/4A/BF/wKioL1QmztuyupjXAAQHwKoGp9Y724.jpg "Title =" 3.png" alt = "wkiol1qmztuyupjxaaqhwkogp9y724.jpg"/>

650) This. width = 650; "src =" http://s3.51cto.com/wyfs02/M01/4A/BD/wKiom1QmzsXi-4FmAAPThFBdN8o785.jpg "Title =" 4.png" alt = "wKiom1QmzsXi-4FmAAPThFBdN8o785.jpg"/>

Due to the previous adjustments and verification, the performance of the server itself has been greatly improved, and the running of the agent is basically controlled within minutes to eliminate the performance problem of the puppet server.

Open debug to check the running status of the agent puppetd -- Test -- debug

It can be seen that an error is reported when you manually run the RPM command on the rpm-related command line (due to damage to the local database of the rpm, the system returns to normal after the database is regenerated)

In addition, pay attention to the details. When you manage the service status through puppet's service, it is implemented by determining the return value of the script (that is, the return value of service XXX status ), if the returned value of the script is incorrect, the Service may be started, but puppet still determines the service exception. To use the puppet management service, the service XXX status should return a normal value.

err: /Stage[main]/Hadoop::Hbase95::Install/Service[datanode]/ensure: change from stopped to running failed: Could not start Service[datanode]: Execution of ‘/sbin/service datanode start‘ returned 127:  at /etc/puppet/modules/hadoop/manifests/hbase95.pp:129

A small example of slow puppet operation