Automated O & M tool Fabric-role management and parallel execution

Automated O & M tool Fabric-role management and parallel execution

Note: This document is written in conjunction with the defing-host-lists and parallel execution chapters in the official Fabric documents.

To allow servers with different functions to execute different commands, Fabric's Rolesenv.roledefsThis function,

Role management

The host string matches a single host, but sometimes it is very useful to group hosts. Maybe you have a group of Web servers under Server Load balancer, and you want to update them all, or run a task on all client servers. Roles provides a correct string definition method for host string groups, which can be used to replace the entire host group.

This ing is defined as a dictionary.env.roledefsTo be used, it must be defined in fabfile. The following is a simple example:

from fabric.api import envenv.roledefs['webservers'] = ['www1', 'www2', 'www3']

Becauseenv.roledefsThe default value is null. Maybe you need to re-allocate the value instead of worrying about losing any information (when you modify it, make sure you didn't load any other fabflies ).

from fabric.api import envenv.roledefs = {    'web': ['www1', 'www2', 'www3'],    'dns': ['ns1', 'ns2']}

In addition to the type of the target host in the list,env.roledefsIs effective immediately. When the module loading time is searched and replaced, it is called (and will thus be called when looked up when tasks are run instead of at module load time) [Note: translate them correctly ]. (For example, you can connect to the remote server to obtain the role definition, And do not worry about delay when the fabfile is loaded when called. For example:fab --list).
There are no other requirements for using roles-it is just a convenient way to provide a group of servers.

The Code that has been put into practice and applied in production is deleted as follows after modification:

#!/usr/bin/python envfrom fabric.api import envfrom fabric.api import runfrom fabric.api import rolesfrom fabric.api import execute#from fabric.context_managers import execute#env.user = 'username'env.password = 'password'env.roledefs = {    'test1': ['host1', 'host2'],    'test2': ['host3', 'host4']}@roles('test1') def get_version():    run('cat /etc/issue')@roles('test2')def get_host_name():    run('hostname')def execute_all():   execute(get_version)   execute(get_host_name)

You can useenv.exclude_hostsOr-xThis command line parameter is excluded.

env.exclude_hosts=['host1']

Or

fab -R web1 -x host1,host5 get_version

In this way, the host1 server will be excluded.

Parallel Execution

Fabric is executed serially by default (for more details, see Execution strategy ). This section describes the options for parallel task execution on multiple hosts. You can use the decorator of each task or the global switch of the command line.

Note: This function is available only in versions 1.3 and later.

Fabric 1.x is NOT thread-safe by default (and generally, task functions do not affect each other). This function is implemented through the Python multiprocessing module. It creates a new thread for each task and host combination. Use a sliding window to avoid running too many threads at the same time.

For example, imagine a scenario where you want to update many Web server application code. Once the code is distributed out, the web Service is reloaded immediately (it is easy to roll back when the update fails). The following fabfile can meet the above requirements:

from fabric.api import *def update():    with cd("/srv/django/myapp"):        run("git pull")def reload():    sudo("service apache2 reload")

Serial execution on three web servers is as follows:

$ fab -H web1,web2,web3 update reload

Under normal circumstances, no parallel execution is used, Fabric will execute in the following order:

1. Update on web1
2. Update on web2
3. Update on web3
4. Reload on web1
5. Reload on web2
6. Reload on web3

When parallel execution is used (view P to obtain more details), it becomes like this:

1. Update on web1, web2, and web3
2. Reload on web1, web2, and web3

This is quite profitable-if the 5S runs as well as 2 s reload is spent. Serial execution takes (5 + 2) x 3 = 21s, while parallel execution only takes 1/3 of the time, with an average of about 7 s.

How to use parallel

Because parallel execution affects a task as the smallest unit. This function can be used to control the switch by using parallel and serial modifiers. For example, fabfile is as follows:

From fabric. api import * # parallel execution task @ paralleldef runs_in_parallel (): pass # sequential execution task def runs_serially (): pass

When running in the following way:

$ fab -H host1,host2,host3 runs_in_parallel runs_serially

The result of continuous execution is as follows:

1. Runs_in_parallel on host1, host2, and host3
2. Runs_serially on host1
3. Runs_serially on host2
4. Runs_serially on host3

Command Line Mark

You can use the command line to mark-P or the environment variable env. parallel enforces parallel execution of all tasks. Even so, all tasks with special serial tags will ignore parallel tags and continue serial execution.

For example:

from fabric.api import *def runs_in_parallel():    pass@serialdef runs_serially():    pass

When this is called:

$ fab -H host1,host2,host3 -P runs_in_parallel runs_serially

For example, in the past, runs_in_parallel was executed concurrently and runs_serially was executed serially.

Bubble size (queue pool concept ?)

In the case of a large number of hosts, your local Fabric host may be overwhelmed by running too many Fabric processes, resulting in a high load on the Fabric host. Because of this, you may need to select a moving bubble method to limit the specified number of active Fabric parallel processes.

By default, Fabric is not used, and all hosts run in a parallel pool. You can specifypool_sizeKeyword to rewrite it. Or set it through global-z.

For example, run five hosts at a time:

from fabric.api import *@parallel(pool_size=5)def heavy_task():    # lots of heavy local lifting or lots of IO here    #

Or skip the parameter pool_size, which is replaced by the following:

$ fab -P -z 5 heavy_task

Use Vagrant and Fabric for Integration Testing

Fabric: Python Remote Deployment Tool

This article permanently updates the link address: