Python multi-process programming (1): Preliminary Exploration

1 basic example
A common practice is to provide parameters when creating a process to tell him what to do. In this example, the output "worker" will be printed five times, but it is unclear whether it is the first or not, because every process is competing to access the output stream.
1.1 Different output sequence

[root@localhost pydoc]# ./tmp.pyworker 0worker 1worker 2worker 3worker 4[root@localhost pydoc]# ./tmp.pyworker 0worker 1worker 2worker 4worker 3

1.2 simple code implementation

[root@localhost pydoc]# cat tmp.py#!/usr/bin/env python#coding=utf-8from multiprocessing import Processdef worker(num):  """test python multi process"""  print 'worker',num  returndef main():  jobs = []  for i in range(5):    p = Process(target=worker,args=(i,))    jobs.append(p)    p.start()if __name__=='__main__':  main()

2. Zombie Process
To wait for a process to complete and exit, you can use the join () method. In Linux, when a process ends, the main process must call wait. Otherwise, the process will become a zombie process. Therefore, it is necessary to call the join () method for each Process object, which is equivalent to wait.

[root@localhost pydoc]# cat tmp.py #!/usr/bin/env python#coding=utf-8from multiprocessing import Processdef worker(num):  """test python multi process"""  print 'worker',num  returndef main():  jobs = []  for i in range(5):    p = Process(target=worker,args=(i,))    jobs.append(p)    p.start()  ##to avoid defunct process,you should call join()  for j in jobs:    j.join()if __name__=='__main__':  main()

3. Open-Source Project MTOP
MySQL MTOP is an open-source MySQL enterprise-level monitoring system developed by Python + PHP. The system uses Python to collect and monitor multi-process data, and PHP to display and manage data on the web. If you are interested, refer to its official website www. mtop. cc. Here we will mainly introduce how to use Python to develop multi-process data collection and monitoring.
3.1 start MTOP

[Root @ localhost mysqlmtop] # mtopctl statusmysql mtop is not run! [Root @ localhost mysqlmtop] # mtopctl startnohup: append the output to "nohup. out" mysql mtop start success! [Root @ localhost mysqlmtop] # mtopctl statusmysql mtop is run! [Root @ localhost mysqlmtop] # ps-ef | grep mtoproot 4463 4446 00:00:00 pts/2 vim mtopd. pyroot 7119 1 0 00:00:00 pts/3 00:00:00/bin/bash/usr/local/bin/mtopctl startroot 7121 7119 1 pts/3 python. /mtopd. pyroot 7125 7121 0 00:00:00 pts/3 python. /mtopd. pyroot 7126 7121 0 00:00:00 pts/3 python. /mtopd. pyroot 7129 7121 0 00:00:00 pts/3 python. /mtopd. pyroot 7130 7121 0 00:00:00 pts/3 python. /mtopd. pyroot 7133 7121 0 00:00:00 pts/3 python. /mtopd. pyroot 7134 7121 0 00:00:00 pts/3 python. /mtopd. pyroot 7186 6613 0 00:00:00 pts/3 grep mtop

From the output of ps, we can also clearly know that the parent process with a pid of 7121 generates six child processes.
3.2 MTOP multi-process data collection trunk code
The code is relatively simple and the basic example is clear.

[root@localhost mysqlmtop]# cat mtopd.py #!/bin/env python#coding:utf-8import os, sys, string, time, datetime, traceback;from multiprocessing import Process;import global_functions as func    def job_run(script_name,times):    while True:        os.system("./"+script_name)        time.sleep(int(times))def main():    print("%s: controller started." % (time.strftime('%Y-%m-%d %H:%M:%S', time.localtime()),))    monitor = func.get_option('monitor')    monitor_status = func.get_option('monitor_status')    monitor_replication = func.get_option('monitor_replication')    monitor_process = func.get_option('monitor_process')    alarm = func.get_option('alarm')    frequency_monitor = func.get_option('frequency_monitor')    frequency_alarm = func.get_option('frequency_alarm')    kill_process = func.get_option('kill_process')    joblist = []    if monitor=="1":        if monitor_status=="1":            job = Process(target = job_run, args = ('check_mysql_status_ext.py',frequency_monitor))            joblist.append(job)            job.start()            job = Process(target = job_run, args = ('check_mysql_status.py',frequency_monitor))            joblist.append(job)            job.start()        if monitor_replication=="1":            job = Process(target = job_run, args = ('check_mysql_replication.py',frequency_monitor))            joblist.append(job)            job.start()        if monitor_process=="1":            job = Process(target = job_run, args = ('check_mysql_process.py',4))            joblist.append(job)            job.start()        if alarm=="1":            job = Process(target = job_run, args = ('alarm_mysql.py',frequency_alarm))            joblist.append(job)            job.start()            if kill_process=="1":            job = Process(target = job_run, args = ('admin_mysql_kill_process.py',3))            joblist.append(job)            job.start()        for job in joblist:            job.join();    print("%s: controller finished." % (time.strftime('%Y-%m-%d %H:%M:%S', time.localtime()),))      if __name__ == '__main__':      main()

Good Luck!