[Python network programming] Design and Implementation of daemon background tasks

[Python network programming] Design and Implementation of daemon background tasks

In B/S-based applications, you often need to run tasks in the background, such as sending emails. In some projects, such as firewalls and WAF, the front-end only displays content and various parameter configurations, and the background daemon process is a major concern. Therefore, you may often see cgi calling on the firewall configuration page, but the actual operation is generally not cgi. For example, to execute the shutdown command, their logic is as follows:

(Ps: The frontend interface includes the backend in web development, otherwise there will be no socket)

<喎?http: www.bkjia.com kf ware vc " target="_blank" class="keylink"> VcD48aDE + release/ejrM/xt8C78Me9tci7 + bG + yc + release/LXIw/release + 8 + release/release + release/da1oaM8YnIgLz7Ex7/Jsru/release + mjnaxvdo3osvnz/vPorj4uvPMqMrYu6S9 + scheme + PGgxPsjnus7Ktc/WPGJyIC8 + PC9oMT7TydPa1 + 69/large + large/large = "brush: java; "> class MgrService (win32serviceutil. serviceFramework): "" Usage: 'python topmgr. py install | remove | start | stop | restart' "# service name_svc_name _ =" Mgr "# service display name_svc_display_name _ =" Daemon Mgr "# service description _ svc_description _ = "Daemon Mgr" def _ init _ (self, args): win32serviceutil. serviceFramework. _ init _ (self, args) self. hWaitStop = win32event. createEvent (None, 0, 0, None) def SvcDoRun (self): self. reportServiceStatus (win32service. SERVICE_START_PENDING) INFO ("mgr startings... ") self. reportServiceStatus (win32service. SERVICE_RUNNING) self. start () # Wait for the Service to be stopped INFO ("mgr waitting... ") win32event. waitForSingleObject (self. hWaitStop, win32event. INFINITE) INFO ("mgr end") def SvcStop (self): self. reportServiceStatus (win32service. SERVICE_STOP_PENDING) INFO ("mgr stopping... ") self. stop () INFO ("mgr stopped") # Set the event win32event. setEvent (self. hWaitStop) self. reportServiceStatus (win32service. SERVICE_STOPPED) def start (self): pass def stop (self): passIn this way, services in windows are implemented, that is, services are run on the background without terminals. INFO and other functions are simple record functions and can be ignored directly. To implement our own background program, we only need to inherit MgrService and provide the start and stop methods.
Because we use socket to transmit messages, we need to listen to the port in the start method, wait for the connection, and process the connection. This is very good for everyone. Here I chose a single thread. Based on coroutine, the underlying layer uses the high-performance network library libev (libevent) --- gevent. If you are interested in gevent, take a look at the in-depth analysis of the gevent running process.

class Engine(MgrService):    rbufsize = -1    wbufsize = 0    def start(self):        INFO('wait connection')        self.server = StreamServer((HOST, PORT), self.msg_handle)        self.server.serve_forever()    def msg_handle(self,socket,address):        try:            rfile = socket.makefile('rb', self.rbufsize)            wfile = socket.makefile('wb', self.wbufsize)            headers = Message(rfile).dict            INFO('get a connection from:%s,headers:%s' % (str(address), headers))            if 'module' in headers and headers['module'] in MODULES:                MODULES[headers['module']].handle(wfile, headers)        except Exception:            ERROR('msg_handle exception,please check')    def stop(self):        if hasattr(self, server):            self.server.stop()

When a new connection arrives, msg_handle processes the message and reads the sent message first. The message format adopts the simplest http format (key name: key value, you have to ask why I use this format. Haha, the format is simple, and python has a ready-made library for parsing.
Considering that there may be many modules in the future, our processing process automatically calls the handle method of the corresponding module based on the module parameters of the message.
The MODULES in the code above is a global variable. When you add a module, You need to register it in MODULES. I provide the module_register method.

MODULES = {           # module: handle module class}def module_register(module_name, handle_class):    if module_name in MODULES:        WARN('duplicate module_name:' + module_name)    else:        MODULES[module_name] = handle_class

Everything is natural here, but it seems that the module only has the handle method. Writing a module by yourself is still very troublesome. You need to think about how to call it and what format of data is most returned, this is a headache, so it is best to provide a base class module.

class Module(object):    SECRE_KEY = "YI-LUO-KEHAN"    MODULE_NAME = "BASE_MODULE"    PREFIX = "do_"  # method prefix    def __init__(self, wfile, headers):        self.wfile = wfile        self.headers = headers    def __getattr__(self, name):        try:            return self.headers[name]        except Exception:            ERROR("%s has no attr:%s,please check" %(self.MODULE_NAME, name))                @classmethod    def handle(cls, wfile, headers):        module_obj = cls(wfile, headers)        module_obj.schedule_default()    def verify(self):        if hmac.new(self.SECRE_KEY, self.MODULE_NAME).hexdigest() == self.signature:            return True        else:            WARN("client verify failed,signature:%s" % str(self.signature))    def schedule_default(self):        err_code = 0        if self.verify() and self.action:            func_name = self.PREFIX + self.action            try:                getattr(self, func_name)()            except AttributeError:                err_code = 1                ERROR("%s has no method:%s" %(self.MODULE_NAME, func_name))            except Exception:                err_code = 2                ERROR("module:%s,method:%s,exception" % (self.MODULE_NAME, func_name))                      else:            err_code = 3        if err_code:            self.send_error({'err_code':err_code})    def send_success(self, msg=''):        data = {'success':True,'msg':msg}        self.wfile.write(json.dumps(data))    def send_error(self, msg=''):        data = {'success':False,'msg':msg}        self.wfile.write(json.dumps(data))

In the base class module, we provide the default processing flow, namely, calling the do_action method based on the action in the message, and providing a simple but effective authentication method, the signature field of a message may be a little simple, but it doesn't matter. You can define your own authentication method.
Next we should write our own modules,

TASK = {}  # task_id: pidclass ScanModule(Module):    MODULE_NAME = "SCAN_MODULE"    def do_start(self):        self.send_success('start ok')        DEBUG('------------task start------------')        task_ids = [int(task_id) for task_id in self.task_ids.split(',') if int(task_id) not in TASK]        for task_id in task_ids:            try:                cmd = 'python scan.py -t %s' % task_id                DEBUG(cmd)                self.sub = Popen(cmd, shell=True, cwd=CWD)                pid = int(self.sub.pid)                TASK[task_id] = pid                INFO('%s start a new task,task_id:%s,pid:%s' %(self.MODULE_NAME, task_id, pid))            except Exception:                ERROR('%s start a new task,task_id:%s failed' % (self.MODULE_NAME, task_id))    def do_stop(self):        self.send_success('stop ok')        DEBUG('------------task stop------------')        task_ids = [int(task_id) for task_id in self.task_ids.split(',') if int(task_id) in TASK]        for task_id in task_ids:            pid = TASK.pop(task_id)            try:                INFO('%s stop a new task,task_id:%s,pid:%s' %(self.MODULE_NAME, task_id, pid))                call(['taskkill', '/F', '/T', '/PID', str(pid)])            except Exception:                ERROR('%s taskkill a task failed,task_id:%s,pid:%s' %(self.MODULE_NAME, task_id, pid))module_register(ScanModule.MODULE_NAME, ScanModule)

The preceding Implementation of a simple scanning module supports two actions, start and stop. Start is very simple. Call the subprocess. Popen of gevent to run the sub-process and record the pid. stop will use taskkill to directly kill the process. Note the following two points: 1. do not use the native subprocess module, because the native subprocess is blocked, which may cause the main processing logic to be blocked. Do not forget to call module_register to register the corresponding module after more requests cannot be served. 2. The method is best to return the result at the beginning, because the foreground is likely to be waiting for the return. So as soon as possible
The following provides a client for testing, client. py

#!/usr/bin/env python#-*-encoding:UTF-8-*-import hmacimport geventfrom gevent import monkeymonkey.patch_socket()addr = ('localhost', 6667)def send_request(module_name,request_headers):    SECRE_KEY = "YI-LUO-KEHAN"    socket = gevent.socket.socket()    socket.connect(addr)    request_headers['module'] = module_name    request_headers['signature'] = hmac.new(SECRE_KEY, module_name).hexdigest()    h = ["%s:%s" %(k, v) for k,v in request_headers.iteritems()]    h.append('\n')    request = '\n'.join(h)    socket.send(request)    print socket.recv(8192)    socket.close()if __name__ =="__main__":    import sys    if sys.argv[1] == 'start':        send_request('SCAN_MODULE',{'action':'start','task_ids':'1'})    else:        send_request('SCAN_MODULE',{'action':'stop','task_ids':'1'})        

Let's perform a simple test: Note: to register a service, cmd requires the administrator privilege. For scan. py called in start, you can simply write one.
As shown in the following figure !!!

The code for this article has been put on github. For more information, see https://github.com/skycrab/pymgr.