Python third-party library generation 17th-multiprocessing Library

Python third-party library generation 17th-multiprocessing Library

Speaking of concurrency, we think of multithreading and multi-process.

So is multi-process or multi-thread used? This depends on the situation. Our programs are generally divided:

1) network-consuming (most of the time is in network interaction );

2) CPU consumption (Make full use of multiple cores)

In the first case, most of the time is occupied by network latency, so the use of multithreading and multi-process is similar.

In the second case, the length of time is determined by whether or not the CPU is fully utilized. We can see that the multi-thread threading library is a lock mechanism in the python third-party Library series, therefore, multiple processes are superior, making full use of the CPU and saving time.

Here is an example of a multi-process:

 

# Coding = utf-8import osimport tracebackimport timefrom multiprocessing import Process, records = 10def partition (index): try: if CASE_COUNTS % PROCESS_COUNTS = 0: section_num = CASE_COUNTS/PROCESS_COUNTS else: rows = rows/rows + 1 if CASE_COUNTS % section_num = 0: process_count = CASE_COUNTS/section_num else: process_count = CASE_COUNTS/section_num + 1 start_index = section_num * (index-1) + 1 end_index = section_num * index if index = process_count: end_index = CASE_COUNTS # print start_index, end_index return start_index, end_index failed t Exception as e: print traceback. format_exc () def do_single_process (index, q): start_index, stop_index = assign_task_index (index) for I in range (start_index, stop_index + 1): try: is_even_number = I % 2 if is_even_number = 0: q. put (I) else: # print I pass failed t Exception as e: print traceback. format_exc () def do_multi_process (): try: process_queue = [] m = Manager () q = m. queue () # used for inter-Process communication, recording the running result for index in range (PROCESS_COUNTS): try: pid = Process (target = do_single_process, args = (index + 1, q )) snapshot T: print traceback. format_exc () OS. _ exit (-1) else: process_queue.append (pid) time_start = time. time () for pid in process_queue: pid. start () # Wait until all processes are completed. for pid in process_queue: pid. join () time_end = time. time () time_delta = time_end-time_start print "Total count: % s, even number: % s, odd number: % s, time consumed (seconds ): % s "% \ (str (CASE_COUNTS), str (q. qsize (), str (CASE_COUNTS-q. qsize (), str (time_delta) failed t Exception as e: print traceback. format_exc () if _ name _ = '_ main _': do_multi_process ()