"Small Project" Python Development automation operations tools-Bulk operations master

There are a lot of open source automation tools are very useful such as Ansible/salt stack, etc., do not have to reinvent the wheel. Just, a lot of OPS students learn Python, suffer from no small project training, this demo with Python write a bulk operation of the tool, we can try to write, perfect perfect.

1 Thinking Analysis

In operations, the old way to deploy the environment, the online code may need to manually knock commands on the server, patience. So, scripts, automation tools and so on are still necessary. I think a batch operation tool should consider the following points:

(1) Essentially, the command is executed on the remote host and the result is returned.

(2) Do batch. That is, to operate on multiple machines concurrently.

(3) Display the returned results clearly to the user.

Usually open-source host batch management tools have two categories, one is the agent, such as Saltstack, puppet, etc., the other is no agent such as Ansible. Although we do not need to repeat the wheel, but can try to write, one is to deepen the understanding of such software principles, and the other is to practice python. It is recommended that if the server size in 1000 units in a non-agent way can hold, if more than 1000, with the agent will be much better.

Let's take a look at how to achieve this.

2 executing commands on a remote machine

There are at least two ways to execute commands on a remote machine and return results: One is to use the Paramiko module, but you can build machine trust and execute SSH commands from there.

Below I put my encapsulated code, is based on the Paramiko module encapsulation, SSH can be implemented by themselves:

Import Paramikoclass Sshparamiko (object): Err = "argument passwd or Rsafile can not is None" Def __init__ (self, host , port, user, Passwd=none, rsafile=none): self.h = host SELF.P = Port self.u = user SELF.W = P asswd Self.rsa = Rsafile def _connect (self): if Self.w:return self.pwd_connect () elif Self.rsa:return self.rsa_connect () else:raise Connectionerror (SELF.ERR) def _transfer (SE        LF): If Self.w:return self.pwd_transfer () elif Self.rsa:return Self.rsa_transfer () Else:raise Connectionerror (SELF.ERR) def pwd_connect (self): conn = Paramiko. Sshclient () Conn.set_missing_host_key_policy (Paramiko. Autoaddpolicy ()) Conn.connect (self.h, SELF.P, self.u, SELF.W) return conn def rsa_connect (self): p Key = Paramiko. Rsakey.from_private_key_file (SELF.RSA) conn = Paramiko. Sshclient () conn.seT_missing_host_key_policy (Paramiko. Autoaddpolicy ()) Conn.connect (hostname=self.h, PORT=SELF.P, username=self.u, Pkey=pkey) return conn def P Wd_transfer (self): transport = Paramiko. Transport (self.h, SELF.P) transport.connect (username=self.u, PASSWORD=SELF.W) sftp = Paramiko. Sftpclient.from_transport (transport) return SFTP, Transport def rsa_transfer (self): Pkey = Paramiko. Rsakey.from_private_key_file (SELF.RSA) transport = Paramiko. Transport (self.h, SELF.P) transport.connect (username=self.u, pkey=pkey) sftp = Paramiko. Sftpclient.from_transport (transport) return SFTP, Transport def run_cmd (self, cmd): conn = Self._connect ( ) stdin, stdout, stderr = Conn.exec_command (cmd) code = stdout.channel.recv_exit_status () stdout, STD        Err = Stdout.read (), Stderr.read () conn.close () If not Stderr:return code, Stdout.decode () Else:return Code, STDERr.decode () def get_file (self, Remote, local): SFTP, conn = Self._transfer () sftp.get (remote, local)        Conn.close () def put_file (self, local, remote): SFTP, conn = Self._transfer () sftp.put (local, remote) Conn.close ()

Of course, the code can also be refactored, ha. Next we look at the effect:

if __name__ = = ' __main__ ':    h = "My test machine IP"    p = $    u = "My user name"    w = "My password"    obj = Sshparamiko (h, p, u, W) 
   r = Obj.run_cmd ("df-h")    print (r[0])    print (r[1])

The result after execution is:

0Filesystem      Size  used Avail use% mounted on/dev/vda1        40G  3.4G   34G   9%/devtmpfs        3.9G     0  3.9G   0%/devtmpfs           3.9G     0  3.9G   0%/dev/shmtmpfs           3.9G  410M  3.5G  11%/runtmpfs           3.9G     0  3.9G   0%/sys/fs/cgroup/dev/vdb        300G   12G  289G   4%/search/odintmpfs           783M     0  783M   0%/run/user/0tmpfs           783M     0  783M   0%/run/user/1000

You can see clearly that the first line is the status code of the command execution, 0 indicates success, not 0 means failure, and the second line starts with the result of our command return. Is it more clear?

3 Concurrent execution and presentation of output results

The

Concurrently executes the thread module that usually comes with Python3, and here I use the From concurrent.futures import Threadpoolexecutor. And when I get the results, I also do some formatting output, such as the green output indicates success, red output indicates that the command failed to execute, yellow indicates a reminder, etc. Don't say much nonsense, just look at the code!

From concurrent.futures import Threadpoolexecutorclass Allrun (object): Def __init__ (self, SSH_OBJS, Cmds, max_worker=50 ): Self.objs = [o for o in ssh_objs] Self.cmds = [C for C in Cmds] Self.max_worker = max_worker # max Number of concurrent threads self.success_hosts = [] # Number of successful machines self.failed_hosts = [] # machine IP Self.mode = failed to store        None Self.func = None def serial_exec (self, obj): "" "Serial execution command on a single machine and return result to dictionary" "" result = List () For C in self.cmds:r = Obj.run_cmd (c) Result.append ([C, R]) return obj, result def conc            Urrent_run (self): "" "Concurrent Execution" "" Future = Threadpoolexecutor (Self.max_worker) for obj in Self.objs: Try:future.submit (self.serial_exec, obj). Add_done_callback (self.callback) except Exceptio N as Err:err = Self.color_str (err, "Red") print (Err) future.shutdown (wait=true) d EF Callback (Self, future_obj): "" "callback function, processing return result" "" ssh_obj, rlist = Future_obj.result () print (Self.color_str ("{} Execute Detail            : ". Format (ssh_obj.h)," yellow ")) Is_success = True for item in Rlist:cmd, [code, RES] = Item info = f "{cmd} |                Code = = {Code}\nresult:\n{res} "if code! = 0:info = Self.color_str (info," red ") is_success = False If ssh_obj.h not in Self.failed_hosts:self.failed_hosts.append (ss             h_obj.h) Else:info = self.color_str (info, "green") print (info) if is_success: Self.success_hosts.append (ssh_obj.h) if ssh_obj.h in Self.failed_hosts:self.failed_ Hosts.remove (SSH_OBJ.H) def overview (self): "" Displays the total execution result "" "For I in Self.success_hosts:print ( Self.color_str (i, "green")) print ("-" *) for J in Self.failed_hosts:print (Self.color_str (J,"Red") info = "Success hosts {};        Failed hosts {}. " s, f = Len (self.success_hosts), Len (self.failed_hosts) info = self.color_str (Info.format (S, f), "yellow") pr Int (info) @staticmethod def color_str (old, Color=none): "" "Add Color to String" "" If color = = "Red": N        ew = "\033[31;1m{}\033[0m". Format (old) elif color = = "Yellow": New = ' \033[33;1m{}\033[0m '. Format (old) elif color = = "Blue": New = "\033[34;1m{}\033[0m". Format (old) elif color = = "Green": NE w = "\033[36;1m{}\033[0m". Format (old) Else:new = old return newif __name__ = = ' __main__ ': H1  = "adime01.shouji.sjs.ted" P1 = u1 = "Odin" W1 = "* * * * *" h = "10.129.206.97" p = $ u = "root" W = "* * * * *" obj1 = Sshparamiko (H1, P1, U1, w1) obj = Sshparamiko (h, p, u, w) cmds = ["Df-h", "ls"] all_obj = All Run ([Obj1, obj], cmds) All_obj.concurrent_run () All_obj.overview ()

The result of the above code run:

From the results of the implementation, highlighting, fresh and clear. It shows each host's command execution status code, returns results, and finally summarizes the results, how many machines were successful, and how many machines failed.

We can also change the execution of the command, let the command execution fail to see:

Later can also be packaged, the host, password, batch execution of commands written in the configuration file, or in accordance with the need to package as command-line tools, in the daily operation and maintenance work can be appropriate to reduce the cumbersome human-Strike command.

"Small Project" Python Development automation operations tools-Bulk operations master