Python script for cluster detection and management

This article mainly introduces the Python script to implement cluster detection and management functions. This article describes implementation ideas, development tool selection, experience sharing, sample code, and so on, if you need a friend, you can refer to the following scenario: In a production data center, there will be a lot of test machines and production machines (that is, about 30 machines ), because of the chaotic management, it is unclear which machine is used and which machine is not used, which leads to an idea: Use one machine to manage all machines, recording Device Owner, device usage status, and other information .... so Why python is simple enough and supports a wide range of third-party libraries.

Initial ideas

Since I have never been in touch with these things since I just joined the work, I need to make things when I shift to a certain department. (if there is no problem with the project, I have to make things, you can't try it out )...

I wanted to create a simple automated testing tool. However, this general testing method and test case cannot be used for the moment (neither input nor output can be determined...

So what do you do? I often find my colleagues asking 208 who uses it? Who uses 201? That IP is mine !!! Have you unplugged my network cable? 242 which machine is it?

Suddenly, spring is coming. Can I use a system to detect IP addresses and device users, or even run a script or command on a certain device as needed? After talking about this short and poor idea with leader, make sure you can do it. let's get started !!!

Design philosophy

The general idea of the system:

1. to obtain information about all servers, you need to deploy an agent on any server as the node for obtaining information, and regularly send server information data to the management server node.
2. as a comprehensive management node, the server receives and stores information submitted by the agent.
3. for ease of use, web pages are used for presentation.

Development Tool selection

1. development language: python
Python is a simple choice, with a wide array of third-party libraries and no need to create a wheel.
2. Database: mysql
Easy to use
3. webpy: web framework
Easy to get started and easy to deploy
4. bootstrap: front-end framework
Don't worry too much about front-end issues
5. paramiko: python library, which complies with the SSH2 protocol and supports remote server connection through encryption and authentication.
Connect to the agent server via SSH: remotely run commands, transfer files
6. scapy: python library, which can be used to send, sniff, parse, and forge network packets. it is used to scan IP addresses.
7. MySQLdb: Connect to mysql
8. shell and python script interfaces: provides shell script interfaces for others.

Experience Sharing

1. the front-end is a new thing for me. I have never done it before. the transition between page animation effects and script runtime needs to be considered. I started to consider using countdown, but this time is uncontrollable, later, ajax was used to solve this problem.
2. the agent must be automatically deployed on each machine, and the refresh time can be controlled through the server.
3. creating a scalable table is very important, and some important information needs to be written to the disk. when the database fails, you can obtain data from the disk.
4. the database connection times out if no operation is performed for a long time.
......

Project structure -- webpy

1. website. py is the main program of webpy, and url ING is set.
2. model. py is the url ing class of webpy. it processes requests and returns
3. static storage of static resources
4. scripts is used to store the processing script. the name here has some problems.

Connect to database

I used MyQSLdb to connect to mysql. here I didn't use the database interface provided by webpy, but encapsulated a set

Ssh remote connection to the server

Paramiko implements ssh connection, data transmission, command execution, and script

The code is as follows:

Def executecmd (cmd, host, port = 22, user = 'root', passwd = 'root '):
Try:
S = paramiko. SSHClient ()
S. set_missing_host_key_policy (paramiko. AutoAddPolicy ())
S. connect (host, port, user, passwd, timeout = 10)
Failed T Exception as e:
S. close ()
Print e
Print 'connet error ...'
Return

Try:
Stdin,stdout,stderrcmds.exe c_command (cmd)
# Print 'host: % s... '% Host
Res = stdout. readlines ()
Failed T Exception as e:
Print 'exec _ commmand error ...'
S. close ()
Return res

Def executefile (file, host, port = 22, user = 'root', passwd = 'root '):
Try:
S = paramiko. SSHClient ()
S. set_missing_host_key_policy (paramiko. AutoAddPolicy ())
S. connect (host, port, user, passwd, timeout = 5)
T = paramiko. Transport (host, port ))
T. connect (username = user, password = passwd)
Sftp = paramiko. SFTPClient. from_transport (t)
Failed T Exception as e:
S. close ()
Print e
Print 'connet error ...'
Return''

Try:
Filename = OS. path. basename (file)
If filename. find ('. sh')> = 0:
Sftp. put (path + '/' + file, '/tmp/tmp_test.sh ')
Stdin,stdout,stderrcmds.exe c_command ('sh/tmp/tmp_test.sh 2>/dev/Null', timeout = 5)
Else:
Sftp. put (path + '/' + file, '/tmp/tmp_test.py ')
Stdin,stdout,stderrcmds.exe c_command ('Python/tmp/tmp_test.py ', timeout = 5)
Initialize stdin,stdout,stderrcmds.exe c_command ('rm-rf/tmp/tmp_test * 2>/dev/Null ')
Res = stdout. readlines ()
S.exe c_command ('rm-rf/tmp/tmp_test * 2>/dev/Null ')
Failed T Exception as e:
S.exe c_command ('rm-rf/tmp/tmp_test * 2>/dev/Null ')
Print 'timeout error ...'
Print e
Return''
Return res

IP scanning

Use scapy for IP scanning

The code is as follows:

Def pro (ip, cc, handle ):
Global dict
Dst = ip + str (cc)
Packet = IP (dst = dst, ttl = 20)/ICMP ()
Reply = sr1 (packet, timeout = TIMEOUT)
If reply:
Print reply. src, is online'
Tmp = [1, reply. src]
Handle. write (reply. src + '\ n ')
# Handle. write (reply. src + "is online" + "\ n ")

Def main ():
Threads = []
Ip = '1970. 168.1.1'
S = 2
E = 254
F = open ('IP. log', 'w ')
For I in range (s, e ):
T = threading. Thread (target = pro, args = (ip, I, f ))
Threads. append (t)
Print "main Thread begins at", ctime ()
For t in threads:
T. start ()
For t in threads:
T. join ()
Print "main Thread ends at", ctime ()

Batch add ssh-key

The code is as follows:

Home_dir = '/home/xx'
Id_rsa_pub = '% s/. ssh/id_rsa.pub' % home_dir

If not id_rsa_pub:
Print 'id _ rsa. pub Does not exist! '
Sys. exit (0)

File_object = open ('% s/. ssh/config' % home_dir, 'w ')
File_object.write ('stricthostkeychecking no \ n ')
File_object.write ('userknownhostsfile/dev/Null ')
File_object.close ()

Def up_key (host, port, user, passwd ):
Try:
S = paramiko. SSHClient ()
S. set_missing_host_key_policy (paramiko. AutoAddPolicy ())
S. connect (host, port, user, passwd)

T = paramiko. Transport (host, port ))
T. connect (username = user, password = passwd, timeout = 3)
Sftp = paramiko. SFTPClient. from_transport (t)

Print 'create Host: % s. ssh dir ...... '% host
Stdin,stdout,stderrcmds.exe c_command ('mkdir ~ /. Ssh /')
Print 'upload id_rsa.pub to Host: % s... '% host
Sftp. put (id_rsa_pub, "/tmp/temp_key ")
Stdin,stdout,stderrcmds.exe c_command ('cat/tmp/temp_key> ~ /. Ssh/authorized_keys & rm-rf/tmp/temp_key ')
Print 'host: % s @ % s auth success! \ N' % (user, host)
S. close ()
T. close ()
Except t Exception, e:
# Import traceback
# Traceback. print_exc ()
Print 'connect error ...'
Print 'delete' + host + 'from database ...'
Delip (host)
# Delete from mysql ****
Try:
S. close ()
T. close ()
Except t:
Pass