[Python network programming] analysis of the daemon background task design and implementation

In doing based on B/s application. There is often a need to perform tasks in the background, with the simplest example of sending mail. In some projects, such as firewalls, WAF, the foreground is only for displaying content and various parameter configurations. The background daemon is the highlight. Therefore, it is possible to see the call CGI in the firewall configuration page often. But the real thing is not CGI, for example, the execution of the shutdown command, their logic such as the following:

(PS: The front-end interface is usually a backend in web development, otherwise there is no socket to say)

Why are you designing this ?You may wonder why you should design this, I think the reasons are as follows:first of all, like firewalls, etc. are basically executed on the class Linux platform1. Security issues CGI generally also has the WWW permission, but the execution of the key commands need root. So we need to get the daemon to do it.2. Generally similar firewall daemon process is written in C/s + +, which is very convenient to handle in message format. If a message structure is filled out, the background process simply needs to be converted to the defined struct body. It is easy to get the number of passes.

It is not possible to remove the Intermediate cig module. Send a message directly to the daemon? I think it's possible. The focus of this article is also to implement this approach.

How to achieve
because it has been under windows for the near future, our daemon is executed under Windows. But in fact Windows does not have a daemon concept. Corresponding is the concept of service. The PYWIN32 package needs to be installed here.

Class Mgrservice (Win32serviceutil. Serviceframework): "" "Usage: ' Python topmgr.py install|remove|start|stop|restart '" "" #服务名 _svc_name_ = " Mgr "#服务显示名称 _svc_display_name_ =" Daemon Mgr "#服务描写叙述 _svc_description_ =" Daemon Mgr "def __init__ (self, args): Win32serviceutil. Serviceframework.__init__ (self, args) Self.hwaitstop = win32event. CreateEvent (None, 0, 0, none) def svcdorun (self): self. Reportservicestatus (Win32service. service_start_pending) INFO ("Mgr Startting ...") self. Reportservicestatus (Win32service. service_running) Self.start () # waits for service to be stopped by INFO ("Mgr Waitting ...") win32event. WaitForSingleObject (Self.hwaitstop, win32event. INFINITE) INFO ("Mgr End") def svcstop (self): self. Reportservicestatus (Win32service. service_stop_pending) info ("Mgr stopping ...") self.stop () info ("Mgr Stopped") # Set event WI N32event.    SetEvent (Self.hwaitstop)    Self. Reportservicestatus (Win32service. service_stopped) def start: Pass def stop (self): Pass

Very easy. This enables the services in Windows, that is, out of the terminal. Executes in the background.

The info function is just a simple record. can be ignored directly.

We have to implement our own daemon, only need to inherit mgrservice, and provide the Start,stop method can be.

because we pass the message through the socket, we listen to the port in the start method and wait for the connection. Process the connection. Everyone is very good at this. I chose it here. single Thread. Based on the co-process, the underlying uses Libev (libevent)--- gevent This high-performance network library. Children's shoes interested in gevent can look at the deep analysis gevent execution process.

Class Engine (Mgrservice):    rbufsize =-1    wbufsize = 0    def start (self):        INFO (' Wait Connection ')        Self.server = Streamserver ((HOST, PORT), Self.msg_handle)        Self.server.serve_forever ()    def msg_handle (self, socket,address):        try:            rfile = Socket.makefile (' RB ', self.rbufsize)            wfile = Socket.makefile (' WB ', self.wbufsize)            headers = Message (rfile). Dict            INFO (' Get a connection from:%s,headers:%s '% (str (address), Headers))            if ' module ' in Headers and headers[' module ' in MODULES:                modules[headers[' module ']].handle (Wfile, Headers)        except Exception:            ERROR (' Msg_handle exception,please check ')    def Stop (self):        if Hasattr (self, server):            self.server.stop ()

when a new connection arrives, it is processed by Msg_handle, and the message sent is first read. Message format uses the simplest format of HTTP, that is (Key name: Key value) format, you want to ask why I use this format, haha, simple format, Python has ready-made library parsing.
Considering that the latter module is likely to be very many, our processing flow will invoke the handle method of the corresponding module according to the module parameters of the message.
The modules of the above code is a global variable, and when you add a module you need to register to modules, and I provide the Module_register method.

MODULES = {           # module:handle module Class}def module_register (module_name, Handle_class):    if module_name in MODUL ES:        WARN (' duplicate module_name: ' + module_name)    else:        modules[module_name] = Handle_class

here all is very natural, but seems only if the module has handle method, write a module yourself or very troublesome, you need to think about how to call, the most return what format of data, this is a headache, so it is best to provide a base class module.

Class Module (object): Secre_key = "Yi-luo-kehan" module_name = "base_module" PREFIX = "Do_" # method PREFIX D         EF __init__ (self, Wfile, headers): Self.wfile = Wfile self.headers = Headers def __getattr__ (self, name): Try:return Self.headers[name] except Exception:error ("%s has no attr:%s,please CHEC K "% (self. Module_name, NAME)) @classmethod def handle (CLS, wfile, headers): Module_obj = CLS (Wfile, headers ) Module_obj.schedule_default () def verify (self): if Hmac.new (self). Secre_key, self. module_name). Hexdigest () = = Self.signature:return True Else:warn ("Client Verify Failed,sign ature:%s "% str (self.signature)) def schedule_default (self): Err_code = 0 if self.verify () and Self.actio N:func_name = self.               PREFIX + self.action try:getattr (self, Func_name) () except Attributeerror: Err_code = 1 ERROR ('%s has no method:%s '% (self. Module_name, Func_name)) except Exception:err_code = 2 ERROR ("module:%s,method:% S,exception "% (self. Module_name, func_name)) Else:err_code = 3 if Err_code:self.send_error ({' Err_code ': Err_code}) def send_success (self, msg= "): data = {' Success ': True, ' msg ': Msg} self.wfile.writ E (json.dumps (data)) def send_error (self, msg= "): data = {' Success ': False, ' msg ': Msg} self.wfile.write (JSO N.dumps (data))

in the base class module we provide the default processing process, which is to invoke the Do_action method based on the action in the message. and provides a simple but very effective authentication method, through the signature field of the message, may be somewhat rudimentary. But it doesn't matter, you can define your own authentication method.
The following is the writing of our own module,

TASK = {} # Task_id:pidclass Scanmodule (Module): module_name = "Scan_module" def Do_start (self): Self.send_ Success (' Start OK ') DEBUG ('------------task start------------') task_ids = [Int (task_id) for task_id in self . Task_ids.split (', ') if int (task_id) not in task] for task_id in task_ids:try:cmd = ' py                Thon scan.py-t%s '% task_id DEBUG (cmd) self.sub = Popen (cmd, shell=true, CWD=CWD) PID = Int (self.sub.pid) task[task_id] = pid INFO ('%s start a new task,task_id:%s,pid: %s '% (self. Module_name, TASK_ID, PID)) except Exception:error ('%s start a new task,task_id:%s failed '% (s Elf. Module_name, task_id)) def do_stop (self): self.send_success (' Stop OK ') DEBUG ('------------task stop------ ------') task_ids = [Int (task_id) for task_id in Self.task_ids.split (', ') if int (task_id) in task] for Task_ ID in TASK_ids:pid = Task.pop (task_id) try:info ('%s stop a new task,task_id:%s,pid:%s '% (sel                 F.module_name, TASK_ID, PID) call ([' Taskkill ', '/F ', '/T ', '/pid ', str (PID)]) except Exception: ERROR ('%s taskkill a task failed,task_id:%s,pid:%s '% (self). Module_name, TASK_ID, PID) module_register (Scanmodule.module_name, Scanmodule)

the above implements a simple scanning module. Supports two action,start,stop. start is very easy. Call Gevent's subprocess. Popen executes the child process and records Pid,stop uses Taskkill to kill the process directly. here are two points to note:1. Do not use the native subprocess module. Because native subprocess are clogged, this can cause the main processing logic to clog and not serve many other requestsFinally, do not forget to call the Module_register register corresponding module. 2. The best way to start is to return the results. Because the foreground is very likely waiting to return.

So say as soon as possible

The following provides a client for testing. client.py

#!/usr/bin/env python#-*-encoding:utf-8-*-import hmacimport geventfrom gevent import monkeymonkey.patch_socket () addr = (' localhost ', 6667) def send_request (module_name,request_headers):    secre_key = "Yi-luo-kehan"    socket = Gevent.socket.socket ()    socket.connect (addr)    request_headers[' module '] = module_name    request_headers[ ' Signature ' = Hmac.new (Secre_key, module_name). Hexdigest ()    h = ["%s:%s"% (k, v) for k,v in Request_headers.iteritems ()]    h.append (' \ n ')    request = ' \ n '. Join (h)    socket.send (Request)    print socket.recv (8192)    Socket.close () if __name__ = = "__main__":    import sys    if sys.argv[1] = = ' Start ':        send_request (' Scan_ MODULE ', {' action ': ' Start ', ' task_ids ': ' 1 '})    else:        send_request (' Scan_module ', {' action ': ' Stop ', ' task_ IDs ': ' 1 '})        

Let's take a quick look at the test:Note: CMD requires administrator privileges to register to the serviceAs for the scan.py called in Start, write a random one to
For example below, we see success!

！

！

This code has been put into github,https://github.com/skycrab/pymgrinterested in children's shoes can be consulted, please give more comments.

[Python network programming] analysis of the daemon background task design and implementation