Python concurrent programming

I. Process theory 1. What is a process

Process: A process that is in progress or a task. and the responsibility for executing the task is CPU

2. Differences between procedures and processes

The program is just a bunch of code, and process refers to the process of running the program

Example:

The chef makes cakes for a guest, and the chef has:

• Recipes for making cakes
• Ingredients: eggs, flour, water

In this analogy:

• Recipe Correspondence Program
• The chef corresponds to the CPU in the computer
• Raw material for the input data
• Chefs read recipes, take out raw materials, make the sum of a series of actions that correspond to the processes in the computer

This time a guest B, asked to make an egg tart for guest B, because B is more important, so the chef records just what step the cake to do (save process State).

Then start looking at the egg tart recipe and start making the egg tart (switch to the high priority process)

When the egg tart is finished, go back to making the cake and continue to do it from the step he left.

Need to emphasize is: the same program executes two times, that is also two processes, for example, open two QQ, each login different account

3. Concurrency and parallelism

Whether it is parallel or concurrent, in the view of the user is ' simultaneously ' running, whether it is a process or a thread, is only a task, the real work is cpu,cpu to do these tasks, and a CPU can only perform one task at a time

1. Concurrency: Pseudo-parallel, which appears to be running at the same time. Single cpu+ multi-channel technology enables concurrent

Example (single core + multi-channel, implementing concurrent execution of multiple processes):

Zhang San at the same time to do more things: writing, practicing listening, watching the United States, but Zhang San the same time can only do one thing,

So in order to achieve the effect of multiple tasks concurrent execution, you can write a job, then practice listening, and then see a beautiful drama

2. Parallel: Run concurrently, only with multiple CPUs to achieve parallelism

Single-core, multi-channel technology can be used, multiple cores, each core can also use multi-channel technology ( Multi-channel technology is for single-core )

There are four cores, six missions, so that four tasks are executed at the same time, assuming they were assigned to CPU1,CPU2,CPU3,CPU4,

Once task 1 encounters I/O and is forced to interrupt execution, Task 5 takes the cpu1 time slice to execute, which is the multi-channel technology under single core.

And once the I/O for Task 1 is over, the operating system calls it back ( knowing the scheduling of the process, which CPU to run, and what the operating system decides )

May be assigned to any one of the four CPUs to execute

4. Creation of processes (understanding)

But all hardware, need to have the operating system to manage, as long as there is an operating system, there is the concept of process, you need to have a way to create processes, some operating systems for a single application program design, such as the microwave oven controller, once the microwave oven, all processes are already there.

For general-purpose systems (running many applications), there is a need to have the ability to create or revoke processes in the process of running the system, mainly divided into 4 forms to create new processes

1. System initialization (view process Linux with the PS command, Windows with Task Manager, the foreground process is responsible for interacting with the user, the process running in the background is not user-independent, the process that runs in the background and wakes only when needed, called daemons, such as e-mail, Web pages, news, printing)
2. A process in the process of running a child process (such as Nginx open multi-process, os.fork,subprocess. Popen, etc.)
3. User's interactive request, while creating a new process (such as user double-click Storm Movie)
4. Initialization of a batch job (applied only in the mainframe batch system)

Either way, the creation of a new process is created by an already existing process that executes a system call to create the process:

1. In Unix, the system call is: Fork,fork creates a copy that is identical to the parent process, with the same storage image, the same environment string, and the same open file (in the shell interpreter process, the execution of a command creates a child process)
2. In Windows, the system call is: Createprocess,createprocess processes both the creation of the process and the process of loading the correct program into a new one.

About creating child processes, UNIX and Windows

1. The same is true: After a process is created, the parent and child processes have their own different address spaces (the multi-channel technology requires the physical plane to implement the isolation of the memory between processes ), and the modification of any one process in its address space does not affect another process.

2. The difference is that, in Unix, the initial address space of a child process is a copy of the parent process, which indicates that the child process and the parent process can have read-only shared memory areas. However, for Windows systems, the address space of the parent process and the child process is different from the beginning.

5. Termination of the process

1. Normal exit (voluntary, such as the user clicks on the interactive page of the cross, or the program has completed the call to initiate system calls to exit normally, in Linux with exit, in Windows with ExitProcess)

2. Error exiting (voluntary, a.py does not exist in Python a.py)

3. Critical Error (involuntary, execution of illegal instructions, such as referencing non-existent memory, 1/0, etc., can catch the exception, try...except ... ）

4. Killed by other processes (involuntary, e.g. kill-9)

6. Hierarchical structure of processes

Regardless of Unix or Windows, the process has only one parent process, and the difference is:

1. All of the processes in Unix are rooted in the init process and form a tree structure. Parent-child processes collectively form a process group so that when a signal is emitted from the keyboard, the signal is sent to all members of the current keyboard-related process group.
2. In Windows, without the concept of process hierarchy, all processes are of the same status, and the only hint similar to the process hierarchy is that when the process is created, the parent process gets a special token ( called a handle) that can be used to control the child process. But the parent process has the right to pass the handle to the other child processes, so there is no hierarchy.

7. Implementation of Process concurrency

The implementation of the concurrency of the process is that the hardware interrupts a running process, saving all the state at which the process is running, and for this reason, the operating system maintains a table, the process table, which consumes a process table entry (these are also referred to as Process Control blocks)

This table holds important information about the status of the process: program counters, stack pointers, memory allocations, status of all open files, account and dispatch information, and other information that must be saved when a process is turned into a ready or blocked state, ensuring that the process starts again as if it had never been interrupted.

8. Supplement the necessary theoretical basis

The role of an operating system:
1: Hide ugly Complex hardware interface, provide good abstraction interface
2: Manage, schedule processes, and compete for hardware with multiple processes in order
Two multi-channel technology:
1. Background: For a single core, to achieve concurrency
Ps:
Now the mainframe is generally multicore, then each core will use multi-channel technology
There are 4 CPUs, a program running in CPU1 encounters IO blocking and will wait until the end of Io and then reschedule, and it will be dispatched to 4
Any one of the CPUs, which is determined by the operating system scheduling algorithm.
2. Reuse in space: if there is a multi-channel program in memory
3. Time reuse: Multiplexing a CPU time slice
Emphasis: Encounter IO cut, take up CPU time too long also cut, the core is to cut the state of the process before saving, so
To ensure that the next time you switch back, you can continue to run based on where you last cut off.

Python concurrent programming