Python Learning notes-pick-up

Multi-process, process queue,pipe pipeline, process lock, process pool, coprocessor, 5 network modes (blocking IO, nonblocking io, signal-driven Io,io multiplexing, asynchronous IO)

Multi-process

Import multiprocessing

Each process will be started by his parent process

In Windows is Pycharm

Linux is multiprocessing.

Get process ID

Import OS
#获取子进程id
Print (Os.getppid ())
#获取父进程id
Print (Os.getpid ())

Output

process Queue data is not shared

threads in the same process the memory is shared transferring data using the previous thread queue

But the memory is not shared between processes, and the data communication between processes is used

process queue, which is not the same as the thread queue, it is equivalent to copying a copy in another process.

from multiprocessing import Process, Queue

def F (QQ):
#在子进程中给 Process Queue Add data
Qq.put ([All, None, ' Hello '])

if __name__ = = ' __main__ ':
#在父进程中创建一个进程queue队列
q = Queue ()
#创建一个进程对象, the process queue is put in as a parameter
p = Process (target=f, args= (q,))
#启动一个进程
P.start ()
Print (Q.get ())
P.join ()

The output is as follows:

Pipe Piping data is not shared, just a copy

communication between two processes can also be used to pipe pipes similar to scoket a hair a

Manager Process data sharing

fromMultiprocessingImportProcess, Manager
defF (D, L):
"This method is used by the process to execute '
    D[1] =' 1 '
    d[' 2 '] ="Wwwww"
    l=["A","B"]
Print (L,D)

if__name__ = =' __main__ ':
#等于 manager= Manager ()
     withManager () asManager
D = manager.dict ()#创建一个字典用于进程之间的数据共享
        L = manager.list ()#创建一个列表用于进程之间的数据共享
        P_list = []#为了循环join的列表
         forIinchRange (10):#循环10次创建10个进程
            p = Process (Target=f, args= (d, L))#创建进程 Get the dictionary and the list in .
            P.start ()
P_list.append (P)
forResinchP_list:
Res.join ()
Print (d)
Print (L)

Process Lock

The purpose of the process lock existence is to multiple processes print data without confusion

Ensure that each process is exclusive to a single screen at the same time

Process Pool  

the role of the process pool High machine overhead when multi-process The process pool guarantees that at the same time in this code

The number of running processes, to avoid excessive overhead and hanging off

Threads do not have a thread pool because they have very little overhead. Need to be able to use semaphores to write a

Co-process

The process, called a micro-thread, is a user-level lightweight thread that has its own register context and stack

You can switch the task up and down continuously. and independently save the location of the last departure, to avoid The time wasted by serial blocking, resulting in a significant increase in efficiency.

The effect is similar to multi-threading, but different

1. When switching

regardless of process or thread each block switching requires a system call, first let the CPU run the operating system's calling program, and then by the scheduler to determine the running of that thread or process

However, the process is controlled by the program itself, without the overhead of thread switching. and multithreading ratio, the more the number of threads, the advantage of the association is more obvious.

2. Thread Lock (Mutex)

Co-process because there is only one thread, so there is no simultaneous write variable conflict, in the process does not need to lock, only judge the state is good, said the efficiency is much higher than multithreading.

for CPU multi-core can be multi-process + co-processes to fully utilize the multi-core, but also play the high efficiency of the association.

Manual co-process Greenlet

co-process Super Automatic version gevent most commonly used

encapsulates the manual co-process (greenlet), when the IO operation is encountered, it automatically switches to the next method, and when the next method encounters an IO operation, it switches back. This avoids the time wasted by waiting for IO execution.

if the thread normally encounters io for a total of 3 seconds, and the co-process takes only 2 seconds

if the co-process method requires parameters, refer to the following F is the method, followed by the parameter

The simplest crawl page equivalent to download

Using the co-process

Socekt implementation of the co-process

Client

Service side

Event-driven

Today's UI programming is event-driven

Click on a task, such as opening a text. Will put this event into a queue,

Then there is a thread to handle the events in the queue. This avoids blocking caused by the mouse or keyboard.

Linux examples

Memory space is divided into

Kernel space the operating system kernel is running space

separate from the normal application access to the underlying hardware Protected Memory Space

User space

Run the program run space

kernel-Space memory and user-space memory is not shared

in writing Scoket, when sending data, the Discovery program is not sent immediately, but waits until the cache threshold is full. This is because the kernel space needs to copy the data to the user space, the data copy operation brings the CPU and the memory overhead is very big , therefore has the cache. The data is much saved and sent together.

One time IO Access ( read example), the data is copied into the buffer of the operating system kernel before it is copied from the buffer of the operating system kernel to the application's address space. So, when a read operation occurs, it goes through two stages:
1. Wait for data preparation (waiting for the
2. copying data from the kernel to the process (Copying The data from the kernel to the)

because of the above two methods, the following 5 kinds of network modes are produced

blocking io single-threaded, can only maintain one scoket link wait and copy data two phases are blocking states

non-blocking io single-threaded, when the user process issued a REAV operation, if the kernel data is not ready, will not block, but return error.

signal-driven IO

IO multiplexing

features: In the single-line Chengcheng, can be cycled to maintain multiple scoket connections, any one of the data is ready to be returned to the user to copy data in the blocking State

Three modes of IO multiplexing:

Select () supported on almost all platforms, only 1024 Scoke T are maintained by default .

There is also if there are 1000 connections, only the last connection returns data. It can be recycled 9,999 times and is a waste of performance.

Poll () is an over-stage

Epoll () does not support Windows support for Linux

Nigix says asynchronous io is actually io multiplexed epoll mode

kinda if there are 1000 connections, there is a connection with data, the kernel will tell which connection has data, the user directly access.

asynchronous io does not block the user process, the kernel waits for the data to be ready, then copies the data to the user's memory, and when all is done, the kernel sends a signal to the user that the read operation is complete. Without any obstruction.

There is very little dedicated module Asyncio for asynchronous IO

Python Learning notes-pick-up