9 Exception handling operating system process thread queue + production consumer model process synchronization callback function

Exception handling

An exception is a signal that an error occurs while the program is running, and in Python the exception that is triggered by the error is as follows

Types of exceptions:

Attributeerror attempts to access a tree that does not have an object, such as foo.x, but Foo does not have a property xioerror input / output exception; it is basically impossible to open the file Importerror the module or package cannot be introduced is basically a path problem or name error Indentationerror syntax error (subclass); The code is not aligned correctly indexerror the subscript index exceeds the sequence boundary, for example, when X has only three elements, it attempts to access x[5]keyerror Attempting to access a key that does not exist in the dictionary keyboardinterrupt Ctrl+c is pressed nameerror use a variable that has not been assigned to the object SyntaxError Python code is illegal, the code cannot compile (personally think this is a syntax error, Wrong) TypeError the incoming object type is not compliant with the requirements Unboundlocalerror attempts to access a local variable that is not yet set, basically because another global variable with the same name causes you to think that you are accessing it valueerror a value that the caller does not expect , even if the type of the value is correct

Common Exceptions

Exception handling:

Python customizes a type for each exception, and then provides a specific syntax structure for exception handling.

Try :2      instrumented code block except  exception type:4     Try to execute the logic of this position once an exception is detected

Basic Syntax

The exception class can only be used to handle the specified exception condition, and cannot be processed if a non-specified exception occurs.

# exception not caught, program directly error 2  'hello'try:5      Int (S1)except  indexerror as E:7     print E

View Code

Multi-Branch

' Hello ' Try :3     int (s1)except  indexerror as E:5     print(e) except Keyerror as E:7     print(e)except  valueerror as E:9     Print (e)

View Code

Universal anomaly

In Python's exception, there is a universal exception: Exception, which can catch arbitrary exceptions, namely:

' Hello ' Try :3     int (s1)except  Exception as E:5     print(e)

View Code

1. If you use exception uniformly, yes, you can catch all exceptions, but that means you use the same logic to handle all exceptions (the logic here is the block of code that is followed by the current expect)

2. If the effect you want is that we need to customize different processing logic for different exceptions, we need to use multiple branches.

' Hello ' Try :    int (s1)except  indexerror as E:    print(e)except Keyerror as E:     Print (e) except ValueError as E:     Print (e)

Multi-Branch

Conclusion:

Try...except should be added only if some anomalies are unpredictable, and other logic errors should be corrected as much as possible.

Operating system

programmers can not understand all the hardware operation details , the management of these hardware and optimize the use of the work is very tedious, this tedious work is the operating system to do , with him, the programmer from these tedious work, freed up, Just consider the writing of your own application software, the application software directly using the functions provided by the operating system to use the hardware indirectly.

To be concise, the operating system is a control program that coordinates, manages, and controls the hardware and software resources of the computer.

Two major functions of the operating system:

Role one: Provides an abstraction of how the application uses hardware resources

Role two: Managing Hardware Resources

Two: multiplexing

Modern computers or networks are multi-user, multiple users not only share hardware, but also share files, databases and other information, sharing means conflict and disorder.

The operating system is primarily used to

1. Log which program uses what resources

2. Allocation of resource requests

3. Mediate conflicting resource requests for different programs and users.

We can summarize the functions of the above operating systems as: processing requests from multiple (multiple, multi-way) shared (shared-as-reuse) resources originating from several programs, referred to as multiplexing.

1. Reuse of Time

When a resource is reused in time, different programs or users take turns using it, and the first program gets the resource used at the end, in the turn to the second ... The third one ...

For example: There is only one CPU, multiple programs need to run on the CPU, the operating system first distributed the CPU to the first program, the program is running long enough (the length of time by the algorithm of the operating system) or encountered an I/O blocking, the operating system will allocate the CPU to the next program, and so on, Until the first program is again assigned to the CPU and then run again, because the CPU switching speed quickly, to the user's feeling is that these programs are running at the same time, or is concurrent, or pseudo-parallel. As for how resources are used for time reuse, or who should be the next program to run, and how long a task needs to run, these are the work of the operating system.

2. Reuse in Space

Each customer acquires a small subset of resources in a large resource, reducing the time it takes to queue up for resources.

For example, multiple running programs go into memory at the same time, and the hardware layer provides a protection mechanism to ensure that the respective memory is split and controlled by the operating system, which is much more efficient than one program exclusive memory and one queue into memory.

Other resources for space reuse also have disks, and in many systems, one disk saves files for many users at the same time. Allocating disk space and documenting who is using which disk block is a typical task for operating system resource management.

The combination of these two approaches is a multi-channel technique.

Process Threads

For the operating system, a task is a process, such as open a browser is to start a browser process, open a notepad started a Notepad process, open two Notepad started the two Notepad process, open a word started a word process.

Some processes do more than one thing at the same time, such as word, which can be typed, spell-checked, and printed at the same time. Within a process, to do multiple tasks at the same time, you need to run multiple "subtasks" at the same time, and we refer to these "subtasks" in the process as threads (thread).
Because each process has at least one thing to do, a process has at least a single thread. Of course, a complex process such as word can have multiple threads, multiple threads can execute simultaneously, multithreading is performed the same way as multiple processes, and the operating system quickly switches between multiple threads, allowing each thread to run briefly alternately, seemingly as if it were executing concurrently. Of course, a multi-core CPU is required to actually execute multiple threads at the same time.
All of the Python programs we wrote earlier are those that perform single-task processes, that is, only one thread. What if we want to do multiple tasks at the same time?

There are two types of solutions:

One is to start multiple processes, although each process has only one thread, but multiple processes can perform multiple tasks in one piece.
Another way is to start a process that starts multiple threads within a process, so that multiple threads can perform multiple tasks in one piece.
Of course, there is a third way, that is, to start multiple processes, each process to start more than one thread, so that the simultaneous execution of more tasks, of course, this model is more complex, and rarely used.
To summarize, there are 3 ways to implement a multitasking:

? Multi-process mode;
? multithreaded mode;
? Multi-process + multithreaded mode.
Performing multiple tasks at the same time is usually not unrelated to each task, but requires communication and coordination with each other, sometimes task 1 must pause waiting for task 2 to finish before it can continue, and sometimes task 3 and task 4 cannot be executed at the same time, so The complexity of multi-process and multi-threaded programs is much higher than the one-process single-threaded program previously written.

There are a lot of times when there's no multitasking. If you want to see a movie on your computer, you have to play the video by one thread, another thread plays the audio, otherwise, the single-threaded implementation can only play the video before playing the audio, or play the audio before playing the video, which is obviously not possible.

Python supports both multi-process and multi-threaded.

Summarize:
A thread is the smallest execution unit, and a process consists of at least one thread. How to schedule processes and threads is entirely up to the operating system, and the program itself cannot decide when to execute and how long it takes to execute.
Multi-process and multi-threaded programs involve synchronization, data sharing problems, and are more complex to write.

Concurrency and parallelism

Whether it is parallel or concurrent, in the view of the user is ' simultaneously ' running, whether it is a process or a thread, is just a task, really work is cpu,cpu to do these tasks, and a CPU can only perform one task at a time

Parallel: Run concurrently, only with multiple CPUs to achieve parallelism

Concurrency: Pseudo-parallel, which appears to be running at the same time. Concurrency can be achieved with a single cpu+ multi-channel technology (concurrent concurrency)

Thread

In traditional operating systems, each process has an address space, and there is a control thread by default

The concept of multithreading (that is, multiple control threads) is that multiple control threads exist in one process, and multiple control threads share the process's address space

　　processes are simply used to bring resources together (a process is just a resource unit, or a collection of resources), and the thread is the executing unit on the CPU ,

For example, the Beijing Metro and the Shanghai Metro are different processes, while the Beijing metro line Line 13 is a thread, the Beijing Metro all the lines share all the resources of the Beijing Metro, such as all passengers can be pulled by all lines.

Why use Multithreading

Multithreading refers to the opening of multiple threads in a process, simply speaking: If multiple tasks share a single address space, then multiple threads must be opened within a process. The detailed lecture is divided into 4 points:

1. Multithreading share the address space of a process

2. Threads are more lightweight than processes, threads are easier to create revocable than processes, and in many operating systems, creating a line turndown creates a process 10-100 times faster, which is useful when a large number of threads require dynamic and rapid modification

3. If multiple threads are CPU-intensive, there is no performance gain, but if there is a lot of computation and a lot of I/O processing, having multiple threads allows these activities to overlap each other, which speeds up the execution of the program.

4. In multi-CPU system, in order to maximize the use of multicore, you can open multiple threads (much less than the open process overhead)

Examples of multithreaded applications:

Open a word processing software process, the process must do more than one thing, such as listening to keyboard input, processing text, automatically save the text to the hard disk, the three tasks are the same piece of data, and therefore can not be used multi-process. Only in one process can open three threads concurrently, if it is a single thread, it can only be, keyboard input, can not handle text and auto-save, automatically save the text can not be entered and processed.

Python concurrent programming Multi-process Multiprocessing module Introduction

Multithreading in Python does not take advantage of multicore advantages, and if you want to fully use the resources of multicore CPUs (Os.cpu_count () Viewing), most of the situations in Python require multiple processes to be used. Python provides a very useful multi-process package multiprocessing.
The multiprocessing module is used to open sub-processes and perform our custom tasks (such as functions) in the subprocess, which is similar to the programming interface of the Multithreaded module threading.

The multiprocessing module has many functions: supporting sub-processes, communicating and sharing data, performing different forms of synchronization, and providing components such as process, Queue, Pipe, lock, etc.

One thing that needs to be emphasized again is that, unlike threads, processes do not have any shared state, the process modifies data, and changes are limited to that process.

Introduction to the Process class

to create a process class :

process ([group [, Target [, name [, args [, Kwargs]]]), an object instantiated by the class that represents a task in a child process (not yet started) emphasizes:1need to use keywords to specify parameters2. args Specifies the positional parameter passed to the target function, which is a tuple form and must have a comma parameter description:1The group parameter is not used and the value is always none2 3target represents the calling object, which is the task to be performed by the child process4 5 args represents the positional parameter tuple of the calling object, Args= (1, 2,'Egon',)6 7 Kwargs represents the dictionary of the calling object, kwargs={'name':'Egon',' Age': 18}8 9 name for child process

Method Description:

1 p.start (): Starts the process and invokes the P.run ()  2 P.run () in the subprocess: The method that runs at the start of the process, which is exactly what it calls the function specified by the target, we must implement the method   in the class of our custom class 3  456  7 p.join ([timeout]): The main thread waits for p to terminate (emphasis: is the main thread is in the state, and P is in the running state). Timeout is an optional time-out, and it should be emphasized that the p.join can only join the START process and not join the run-open process  

Property Description:

1 P.daemon: The default value is False, if set to true, for p to run in the background daemon, when P's parent process terminates, p also terminates, and set to True, p cannot create its own new process, must be set before P.start () 2 3 p.name: Name of the process 4 5 p.pid: PID of theprocess 6 7 P.exitcode: The process is none at run time, if it is –n, the signal N ends (understand)8 9 P.authkey: The authentication key for the process, which by default is a 32-character string that is randomly generated by os.urandom (). The purpose of this key is to provide security for the underlying interprocess communication that involves a network connection, which can only succeed if you have the same authentication key (understand it)

Use of the Process class

=====================Part1: Two ways to create and open child processes

Note: the process () in Windows must be placed in # if __name__ = = ' __main__ ': under

Since Windows has no fork, the multiprocessing module starts a new Python process and imports the calling module.
If Process () gets called upon import, then this sets off a infinite succession of new processes (or until your machine RU NS out of resources).
The reason for hiding calls to Process () inside

if __name__ = = "__main__"
Since statements inside this if-statement would not get called upon import.

Because Windows does not have fork, the multi-processing module launches a new Python process and imports the calling module.
If you call process () on import, this will start a new process with infinite inheritance (or until the machine runs out of resources).
This is the original to hide the internal call to process (), using if __name__ = = "__main __", the statement in this if statement will not be called at the time of import.

#method One of the open process:Import TimeImportRandom fromMultiprocessingImportProcessdefPiao (name):Print('%s piaoing'%name) time.sleep (Random.randrange (1,5))    Print('%s Piao End'%name) P1=process (target=piao,args= ('Egon',))#must be added, No.P2=process (target=piao,args= ('Alex',)) P3=process (target=piao,args= ('Wupeqi',)) P4=process (target=piao,args= ('Yuanhao',)) P1.start () P2.start () P3.start () P4.start ( )Print('Main Thread')

method One of the open process:

#two ways to open a process:Import TimeImportRandom fromMultiprocessingImportProcessclassPiao (Process):def __init__(Self,name): Super ().__init__() Self.name=namedefRun (self):Print('%s piaoing'%self.name) Time.sleep (Random.randrange (1,5))        Print('%s Piao End'%self.name) P1=piao ('Egon') P2=piao ('Alex') P3=piao ('Wupeiqi') P4=piao ('Yuanhao') P1.start ()#start will automatically call runP2.start () P3.start () P4.start ( )Print('Main Thread')

method of open process two

Practice one: Turn the socket communication learned last week into a form of concurrency

 fromSocketImport* fromMultiprocessingImportProcessserver=socket (af_inet,sock_stream) server.setsockopt (sol_socket,so_reuseaddr,1) Server.bind ('127.0.0.1', 8080)) Server.listen (5)defTalk (conn,client_addr): whileTrue:Try: Msg=CONN.RECV (1024)            if  notMsg BreakConn.send (Msg.upper ())exceptException: Breakif __name__=='__main__':#The START process under Windows must be written down here.     whiletrue:conn,client_addr=server.accept () p=process (target=talk,args=(CONN,CLIENT_ADDR)) P.start ()

Server Side

 fromSocketImport*Client=socket (Af_inet,sock_stream) client.connect (('127.0.0.1', 8080)) whiletrue:msg=input ('>>:'). Strip ()if  notMsgContinueclient.send (Msg.encode ('Utf-8')) msg=CLIENT.RECV (1024)    Print(Msg.decode ('Utf-8'))

multiple Client Side

Each client, all on the server to open a process, if the concurrent to a million client, to open 10,000 processes, you yourself try to open 10,000 on your own machine, 100,000 processes to try. Workaround: Process Pool

9 Exception handling operating system process thread queue + production consumer model process synchronization callback function