Python concurrent programming instance tutorial, python instance tutorial
Python concurrent programming example tutorial
The example of concurrent programming in Python is mainly used for the Threading module. A Threading class library is defined in this document.
I. Introduction
We call a running program a process. Each process has its own system status, including the memory status, open file list, program pointer tracking command execution, and a call stack that saves local variables. Generally, a process is executed in a single sequence of control flows. This control flow is called the main thread of the process. At any given moment, a program only does one thing.
A program can create a new process (such as OS. fork () or subprocess. Popen () through the OS or subprocess module in the Python library function ()). However, these processes, called sub-processes, run independently. They have their own independent system states and main threads. Because processes are independent from each other, they are concurrently executed with the original process. This means that the original process can execute other work after creating the sub-process.
Although processes are independent of each other, they can communicate with each other through the mechanism called inter-process communication (IPC. A typical mode is based on message transmission, which can be simply understood as a pure byte buffer, while send () or recv () operation primitives can be used through pipelines such as pipe) or I/O channels such as network socket to transmit or receive messages. There are also some IPC modes that can be completed through the memory-mapped mechanism (for example, the mmap Module). Through memory ing, the process can create shared areas in the memory, modifications to these regions are visible to all processes.
Multi-process can be used in scenarios where multiple tasks need to be executed simultaneously. Different processes are responsible for different parts of the task. However, another way to segment jobs into tasks is to use threads. Similar to a process, a thread also has its own control flow and execution stack, but the thread runs within the process it was created to share all the data and system resources of its parent process. Threads are useful when applications need to complete concurrent tasks, but the potential problem is that tasks must share a large number of system states.
When multiple processes or threads are used, the operating system is responsible for scheduling. This is achieved by a small time slice for each process (or thread) and fast cyclic switching between all active tasks. This process divides the CPU time into small fragments and distributes them to each task. For example, if 10 active processes in your system are being executed, the operating system will allocate a CPU time to each process and switch among the 10 processes cyclically. When the system has more than one CPU core, the operating system can schedule processes to different CPU cores to maintain the average system load for parallel execution.
Complex issues need to be considered for programs written using the concurrent execution mechanism. The main source of complexity is data synchronization and sharing. In general, multiple tasks attempt to update the same data structure at the same time, which may result in inconsistent dirty data and program status (formally speaking, it is about resource competition ). To solve this problem, you need to use mutex or other similar synchronization primitives to identify and protect key parts of the program. For example, if multiple different threads are attempting to write data to the same file at the same time, you need a mutex lock to execute these write operations in sequence. When a thread is writing data, other threads must wait until the current thread releases the resource.
Concurrent Programming in Python
Python has long supported concurrent programming in different ways, including threads, sub-processes, and other concurrent implementations using the generator function.
Python supports message passing and thread-based concurrent programming in most systems. Although most programmers are more familiar with the thread interface, the Python thread mechanism has many restrictions. Python uses the internal global interpreter lock (GIL) to ensure thread security. GIL allows only one thread to execute at the same time. This allows the Python program to run on a single processor even on a multi-core system. Despite a lot of arguments in the Python field about GIL, there is no possibility of removing it in the foreseeable future.
Python provides some sophisticated tools for managing concurrent operations based on threads and processes. Even a simple program can use these tools to speed up concurrent tasks. The subprocess module provides APIs for creating and communicating sub-processes. This is especially suitable for running text-related programs because these Apis support data transmission through standard input/output channels of new processes. The signal module exposes the semaphore mechanism of the UNIX System to users to transmit event information between processes. Signals are processed asynchronously. When a signal arrives, the current work of the program is interrupted. Signal mechanisms can implement coarse-grained message transmission systems, but other more reliable in-process communication technologies can transmit more complex messages. The threading module provides a series of advanced object-oriented APIs for concurrent operations. Thread objects run concurrently in a process to share memory resources. (Www.jbxue.com) Use threads to better expand I/O-intensive tasks. The multiprocessing module is similar to the threading module, but it provides process operations. Each process class is a real operating system process and does not share memory resources. However, the multiprocessing module provides a mechanism to share data between processes and transmit messages. Generally, it is easy to change a thread-based program to process-based. You only need to modify some import declarations.
Threading module example
Taking the threading module as an example, let's think about a simple question: how to accumulate a large number using the multipart parallel method.
Import threading
Class SummingThread (threading. Thread ):
Def _ init _ (self, low, high ):
Super (SummingThread, self). _ init __()
Self. low = low
Self. high = high
Self. total = 0
Def run (self ):
For I in range (self. low, self. high ):
Self. total + = I
Thread1 = SummingThread (0, 500000)
Thread2 = SummingThread (500000,100 0000)
Thread1.start () # This actually causes the thread to run
Thread2.start ()
Thread1.join () # This waits until the thread has completed
Thread2.join ()
# At this point, both threads have completed
Result = thread1.total + thread2.total
Print (result)
Custom Threading class library
I wrote a small Python class library that is easy to use with threads, including some useful classes and functions.
Key parameters:
* Do_threaded_work-this function assigns a series of given tasks to corresponding processing functions (the allocation order is uncertain)
* ThreadedWorker-This class creates a thread that pulls a job from a synchronous work queue and writes the processing result to the synchronous result queue.
* Start_logging_with_thread_info-Write the thread id to all log messages. (Dependent on the log environment)
* Stop_logging_with_thread_info-used to remove the thread id from all log messages. (Dependent on the log environment)
Import threading
Import logging
Def do_threaded_work (work_items, work_func, num_threads = None, per_sync_timeout = 1, preserve_result_ordering = True ):
"Executes work_func on each work_item. Note: Execution order is not preserved, but output ordering is (optionally ).
Parameters:
-Num_threads Default: len (work_items) --- Number of threads to use process items in work_items.
-Per_sync_timeout Default: 1 --- Each synchronized operation can optionally timeout.
-Preserve_result_ordering Default: True --- Reorders result_item to match original work_items ordering.
Return:
--- List of results from applying work_func to each work_item. Order is optionally preserved.
Example:
Def process_url (url ):
# TODO: Do some work with the url
Return url
Urls_to_process = ["http://url1.com", "http://url2.com", "http://site1.com", "http://site2.com"]
# Process urls in parallel
Result_items = do_threaded_work (urls_to_process, process_url)
# Print (results)
Print (repr (result_items ))
"""
Global wrapped_work_func
If not num_threads:
Num_threads = len (work_items)
Work_queue = Queue. Queue ()
Result_queue = Queue. Queue ()
Index = 0
For work_item in work_items:
If preserve_result_ordering:
Work_queue.put (index, work_item ))
Else:
Work_queue.put (work_item)
Index + = 1
If preserve_result_ordering:
Wrapped_work_func = lambda work_item: (work_item [0], work_func (work_item [1])
Start_logging_with_thread_info ()
# Spawn a pool of threads, and pass them queue instance
For _ in range (num_threads ):
If preserve_result_ordering:
T = ThreadedWorker (work_queue, result_queue, work_func = wrapped_work_func, queue_timeout = per_sync_timeout)
Else:
T = ThreadedWorker (work_queue, result_queue, work_func = work_func, queue_timeout = per_sync_timeout)
T. setDaemon (True)
T. start ()
Work_queue.join ()
Stop_logging_with_thread_info ()
Logging.info ('work _ queue joined ')
Result_items = []
While not result_queue.empty ():
Result = result_queue.get (timeout = per_sync_timeout)
Logging.info ('found result [: 500]: '+ repr (result) [: 500])
If result:
Result_items.append (result)
If preserve_result_ordering:
Result_items = [work_item for index, work_item in result_items]
Return result_items
Class ThreadedWorker (threading. Thread ):
"Generic Threaded Worker
Input to work_func: item from work_queue
Example usage:
Import Queue
Urls_to_process = ["http://url1.com", "http://url2.com", "http://site1.com", "http://site2.com"]
Work_queue = Queue. Queue ()
Result_queue = Queue. Queue ()
Def process_url (url ):
# TODO: Do some work with the url
Return url
Def main ():
# Spawn a pool of threads, and pass them queue instance
For I in range (3 ):
T = ThreadedWorker (work_queue, result_queue, work_func = process_url)
T. setDaemon (True)
T. start ()
# Populate queue with data
For url in urls_to_process:
Work_queue.put (url)
# Wait on the queue until everything has been processed
Work_queue.join ()
# Print results
Print repr (result_queue)
Main ()
"""
Def _ init _ (self, work_queue, result_queue, work_func, stop_when_work_queue_empty = True, queue_timeout = 1 ):
Threading. Thread. _ init _ (self)
Self. work_queue = work_queue
Self. result_queue = result_queue
Self. work_func = work_func
Self. stop_when_work_queue_empty = stop_when_work_queue_empty
Self. queue_timeout = queue_timeout
Def should_continue_running (self ):
If self. stop_when_work_queue_empty:
Return not self. work_queue.empty ()
Else:
Return True
Def run (self ):
While self. should_continue_running ():
Try:
# Grabs item from work_queue
Work_item = self. work_queue.get (timeout = self. queue_timeout)
# Works on item
Work_result = self. work_func (work_item)
# Place work_result into result_queue
Self. result_queue.put (work_result, timeout = self. queue_timeout)
Failed t Queue. Empty:
Logging. warning ('threadedworker Queue was empty or Queue. get () timed out ')
Failed t Queue. Full:
Logging. warning ('threadedworker Queue was full or Queue. put () timed out ')
Except t:
Logging. exception ('error in threadedworker ')
Finally:
# Signals to work_queue that item is done
Self. work_queue.task_done ()
Def start_logging_with_thread_info ():
Try:
Formatter = logging. Formatter ('[thread % (thread)-3 s] % (message) s ')
Logging. getLogger (). handlers [0]. setFormatter (formatter)
Except t:
Logging. exception ('failed to start logging with thread info ')
Def stop_logging_with_thread_info ():
Try:
Formatter = logging. Formatter ('% (message) s ')
Logging. getLogger (). handlers [0]. setFormatter (formatter)
Except t:
Logging. exception ('failed to stop logging with thread info ')
Example
From test import ThreadedWorker
From queue import Queue
Urls_to_process = ["http://facebook.com", "http://pypix.com"]
Work_queue = Queue ()
Result_queue = Queue ()
Def process_url (url ):
# TODO: Do some work with the url
Return url
Def main ():
# Spawn a pool of threads, and pass them queue instance
For I in range (5 ):
T = ThreadedWorker (work_queue, result_queue, work_func = process_url)
T. setDaemon (True)
T. start ()
# Populate queue with data
For url in urls_to_process:
Work_queue.put (url)
# Wait on the queue until everything has been processed
Work_queue.join ()
# Print results
Print (repr (result_queue ))
Main ()
I want to learn Python programming. I recommend a few tutorials. There are also a few questions about the latest version of python you installed. Most online textbooks are based on Python, you can download a version 2.5 and install it. 3 there are still many problems and many packages are incompatible.
What is the self-learning method of the python Programming Tutorial? Let's take a look at the concise python tutorial and the python core programming.