The 38th day of walking into the computer (Python process and co-formation))

First, the process

1, multiprocessing module to achieve multi-process concurrency.

The 1.1multiprocessing package is a multi-process Management Pack in Python, with threading. Like thread, it can take advantage of multiprocessing. The process object to create a process that can run a function written inside a python program that belongs to the relationship between the parent and child processes. The process object is used in the same way as the thread object, and there is a method for start (), run (), join ().

Company of the 1.2Process class

Construction Method:

Process ([group [, Target [, name [, args [, Kwargs]]])

Group: Thread groups, currently not implemented, the library reference must be none;
Target: The method to be executed;
Name: Process name;
Args/kwargs: The parameter to pass in the method.

Example method:

Is_alive (): Returns whether the process is running.

Join ([timeout]): Blocks the process of the current context until the process calling this method terminates or arrives at the specified timeout (optional parameter).

Start (): Process ready, waiting for CPU scheduling

Run (): Strat () calls the Run method, and if the instance process does not have an incoming target set, this star executes the T default run () method.

Terminate (): Stops the worker process immediately, regardless of whether the task is completed

Property:

Daemon: Same as thread's Setdeamon function

Name: Process name.

PID: Process number.

1.3. Instance operation

　　　　　　

2. Interprocess communication Method-process queue

2.1 Process queues queue, similar to thread queues:

　　　　　　

2.2 Piping (pipe),:

　　　　　　

Note:The two connection objects returned by pipe () represent both ends of the pipeline. Each connection object has the Send () and recv () methods (and so on). Note that if two processes (or threads) try to read or write to the same end of the pipeline at the same time, the data in the pipeline may be corrupted.

2.3manager () data sharing between processes:

　　　　

Note: The parent process must wait until the child process finishes executing to perform the next step.

The queue and pipe only achieve data interaction and do not share data, but the manager is a process to change the data of another process .

3. Process Pool

3.1 What is a process pool?

The process pool maintains a sequence of processes that, when used, gets a sequence of processes in the process pool, and if there are no process sequences available for use in the process pool, the program waits until a sequence of processes is available in the process pool.

3.2:

　　　　　　

There are several main methods in the process pool:

Apply: Take a process from the process pool and execute

Asynchronous version of Apply_async:apply, once taken out and then executed

Terminate: Immediately close the thread pool

Close: Wait for all processes to finish before shutting down the thread pool

Join: The main process waits for all child processes to finish executing and must be after close or terminate

Ii. co-formation

1. What is co-forming?

Co-process, also known as micro-threading, fiber. English name Coroutine, the association is a user-state lightweight thread.

2. Co-forming features

The co-process has its own register context and stack. When the schedule is switched, the register context and stack are saved elsewhere, and the previously saved register context and stack are restored when it is cut back. Thus: The process can retain the state of the last invocation (that is, a specific combination of all local states), each time the procedure is re-entered, which is equivalent to the state of the last call, in other words: The position of the logical flow at the last departure.

3, yield and co-formation,:

　　　　

4, Greenlet realization of co-formation

　　　　

The main idea of the Greenlet mechanism is that the yield statement in the generator function or the co-function will suspend the execution of the function until later using the next () or send () operation to recover. You can use a scheduler loop to collaborate on multiple tasks between a set of generator functions. Greentlet is a basic library of Python that implements what we call "Coroutine".

5, Gevent realization of co-formation

　　　　

Note: Prior to the above two implementations, the Gevent implementation will switch to the next task when the IO operation is encountered, greatly saving the CPU time to execute the program.

Comments:

The benefits of the process: without the overhead of thread context switching, the overhead of atomic operation locking and synchronization, the convenience of switching control flow, simplifying the programming model; high concurrency + high scalability + Low cost: A CPU support for tens of thousands of processes is not a problem. Therefore, it is suitable for high concurrency processing.
Disadvantage: Unable to take advantage of multicore resources: The nature of the process is single-threaded, it can not be a single CPU at the same time multiple cores, the process needs and processes to run on multi-CPU. Of course, most of the applications we write on a daily basis are not necessary, except for CPU-intensive applications. Blocking (Blocking) operations (such as IO) can block the entire program

The 38th day of walking into the computer (Python process and co-formation))