Python automated Transport Koriyuki advanced functions

The concept of the

One, the co-process
1.1 process is
co-process, also known as micro-threading, fiber. English name Coroutine. One sentence explains what a thread is: The process is a lightweight thread that is user-state. (Not really white ~)
So it's easier to understand the process:
threads are system-level, they are scheduled by the operating system, and the process is program-level, which is scheduled by the programmer on demand. We call one of the functions in a thread called subroutines, then the subroutine can be interrupted in the execution of the other subroutine, the other subroutine can also be interrupted back to continue to execute the previous subroutine, this is the process. That is, a piece of code under the same thread execution can be interrupted, and then jump to execute another section of code, when again back to execute the code block, and then from the place where the previous break to start execution. The
more professional understanding is that the
process has its own register context and stack. When the schedule is switched, the register context and stack are saved elsewhere, and the previously saved register context and stack are restored when it is cut back. Thus: The process can retain the state of the last invocation (that is, a specific combination of all local states), each time the procedure is re-entered, which is equivalent to the state of the last call, in other words: The position of the logical flow at the last departure.
1.2 Advantages and disadvantages of the process
the benefits of the process:
(1) without the overhead of thread context switching, the process avoids meaningless scheduling, which can improve performance (but also therefore, the programmer must assume responsibility for the scheduling, while the co-process also loses the ability of the standard thread to use multiple CPUs)
(2) Cost of locking and synchronizing without atomic Operation
(3) facilitates switching of control flow, simplifies programming model
  (4) high concurrency + high scalability + Low cost: One CPU support for tens of thousands of processes is not a problem. Therefore, it is suitable for high concurrency processing.
Disadvantages of the process:
(1) cannot take advantage of multi-core resources: The nature of the process is a single-threaded, it can not be a single CPU at the same time multiple cores, the process needs and processes to run on multi-CPU. Of course, most of the applications we write on a daily basis are not necessary, except for CPU intensive applications.
(2) blocking (Blocking) operations (such as IO) block the entire program
2, how to implement the Coprocessor
2.1 yield implementation coprocessor
earlier in this article, "subroutines (functions) can be interrupted to execute other subroutines during execution. , other subroutines can be interrupted back to continue to execute the previous subroutine ", it is easy to think of the yield of Python, obviously yield is able to achieve this switch.

Def eater (name): Print ("%s eat food"%name) and True:food = yield print ("done") G = Eater ("Gangdan") prin T (g)

Execution Result:

<generator Object Eater at 0x0000000002140fc0>

The result of execution can prove that G is now a generator function

2.2 Process of function assignment
In the form of yield expression, to run next (), let the function initialize and stop at yield, and then send (), send will assign a value to yield when the next execution of the code is triggered
Next () and send () all allow the function to continue running at the last paused position.

def creater (name): Print ('%s start to eat food '%name) food_list = [] while True:food = yield food_list Print ('%s get%s, to start eat '% (Name,food)) food_list.append (food) # get Builder builder = creater (' Tom ') # now is the run function, Let the function initialize the next (builder) print (Builder.send (' Bun ')) print (Builder.send (' bone ')) Print (Builder.send (' Vegetable soup '))

Execution Result:

Tom start to eat Foodtom get bun, to start eat[' bun ']tom get bone, to start eat[' bun ', ' bone ']tom get vegetable soup, to start eat[' bun ', ' bone ' , ' Vegetable soup '

The

Note is that each time you need to run the next () function, the program stays in the yield position.
If more than one such function needs to execute the next () function, the program stays in the yield position. To prevent forgetting to initialize the next operation, you need to use an adorner to resolve the problem

Def init (func):     def wrapper (*args,**kwargs):         builder = func (*args,**kwargs)         next (builder)     #  This place is the key you can use Builder.send ("none") and must pass in none for the first time.         return builder    return  Wrapper@initdef creater (name):     print ('%s start to eat food ')  %name)     food_list = []    while True:         food = yield food_list         print ('%s get %s ,to start eat '  % (name,food))          food_list.append (food) #  Get generator Builder = creater ("Tom") #   is now a direct-running function without the need to initialize print (builder.send (' Bun ')) print (BUILDER.SEnd (' Bones ')) print (Builder.send (' Vegetable soup ')) 

Execution Result:

Tom start to eat Foodtom get bun, to start eat[' bun ']tom get bone, to start eat[' bun ', ' bone ']tom get vegetable soup, to start eat[' bun ', ' bone ' , ' Vegetable soup '

2.3 Simple application of the co-process function
Please feed Tom Food

Def init (func):     def wrapper (*args,**kwargs):         builder = func (*args,**kwargs)         next (builder)         return builder    return  wrapper@initdef creater (name):     print ('%s start to eat  Food '  %name)     food_list = []    while true:         food = yield food_list         print ('%s get %s ,to start eat '  % (name,food))          food_list.append (food) def food ():     Builder = creater ("Tom")     while True:         food = input ("Please feed Tom Food: "). Strip ()         if food == " Q ":             print ("Feed End")              return 0        else:             builder.send (food) if __name__ ==   ' __main__ ':     food ()

Execution Result:

Tom start to eat food please feed Tom: Bones Tom get bone, to start eat please feed Tom food: Vegetable soup tom Get vegetable soup, to start eat please feed Tom Food: Q Feeding End

2.4 Application of the co-process function
Implementation of the "GREP-RL error < directory >" command in Linux, filtering the contents of a file under the sub-file, Word folder content of the function program
First understand the walk method in an OS module to open the folder under the path in the parameter and return a tuple

>>> Import OS # import module >>> os.walk (r "E:\Python\script") #使用r is to let the symbols in the string have no special meaning, for the escape <generator object Walk at 0x00000000035d3f10>>>> g = Os.walk (r "E:\Python\script") >>> Next (g) (' E:\\python\\script ' , ['. Idea ', ' functions '], [])

Returns a tuple, the first element is the path to the file, the second is a folder, and the third is the file under that path
There is a need to write the idea of a program: process-oriented programming
Second, process-oriented programming
Process-oriented: The core is the process of two words, process and the steps to solve the problem, based on process-oriented design program is an industrial pipeline, is a mechanical way of thinking. Pipeline programming idea, in the design process, need to design the entire process
Advantages:
1: Architecture Clearer
2: Simplifying the complexity of the program
Disadvantages:
Scalability is extremely poor, so the process-oriented scenario is: Software that does not require frequent changes, such as: Linux kernel, httpd,git, etc.

The following is a process-oriented concept to complete the function of the application of the function
Directory structure:

Test├──aa│├──bb1││└──file2.txt│└──bb2│└──file3.txt└─file1.txt File Contents: file1.txt:error123file2.txt:1 23file3.txt:123error

Program Flow
First stage: Find the absolute path to all files
Second Stage: Open file
Phase three: Iterate through each line
Phase IV: Filtering "error"
Fifth stage: Print the file name that the line belongs to

First stage: Find the absolute path to all files
G is a generator that can be executed with next (), and each time next is run, the result is the path of the file opened in turn

>>> Import os>>> g = Os.walk (r "E:\Python\script\ function \test") >>> Next (g) (' E:\\python\\script\ \ function \\test ', [' AA '], []] >>> Next (g) (' e:\\python\\script\\ function \\test\\aa ', [' bb1 ', ' bb2 '], [' file1.txt ']) > >> Next (g) (' e:\\python\\script\\ function \\test\\aa\\bb1 ', [], [' File2.txt ']) >>> next (g) (' e:\\python\\ script\\ function \\test\\aa\\bb2 ', [], [' File3.txt ']) >>> next (g) Traceback (most recent call last): File "<input > ", Line 1, in <module>stopiteration

When we open the file, we need to find the absolute path of the file, and now we can stitch the first and third parts with the method of string concatenation.
To open with a loop:

Import Osdir_g = Os.walk (r "E:\Python\script\ function \test") for Dir_path in Dir_g:print (Dir_path)

Results:

(' e:\\python\\script\\ function \\test ', [' AA '], []) (' e:\\python\\script\\ function \\test\\aa ', [' bb1 ', ' bb2 '], [' file1.txt ']) (' e:\\python\\script\\ function \\test\\aa\\bb1 ', [], [' File2.txt ']) (' e:\\python\\script\\ function \\test\\aa\\bb2 ', [], [' File3.txt '])

Stitching the queried files and paths into absolute paths

Import Osdir_g = Os.walk (r "E:\Python\script\ function \test") for Dir_path in dir_g:for file in dir_path[2]: File = "%s\ \%s "% (dir_path[0],file) print (file)

Execution Result:

E:\Python\script\ function \test\aa\file1.txte:\python\script\ function \test\aa\bb1\file2.txte:\python\script\ function \test\aa\bb2\ File3.txt

Implemented with functions:

Import Osdef Search (): While true:dir_name = yield Dir_g = Os.walk (dir_name) for Dir_path in Dir_ G:for file in dir_path[2]: File = "%s\\%s"% (dir_path[0],file) print (file) g = SE Arch () Next (g) G.send (r "E:\Python\script\ function \test")

In order to return the results to the next process

@init # Initialize Generator def search: While true:dir_name = yield Dir_g = Os.walk (dir_name) for par Dir,_,files in dir_g:for file in Files:abspath = r "%s\%s"% (pardir,file) target . Send (Abspath)

Second Stage: Open file

@initdef opener (target): While True:abspath=yield with open (Abspath, ' RB ') as F:target.send (( ABSPATH,F))

Phase three: Iterate through each line of content

@initdef Cat (target): While True:abspath,f=yield # (ABSPATH,F) for line in F:res=target.send (            (Abspath,line)) If Res:break

Stage four: Filtration

@initdef grep (pattern,target): Tag=false while True:abspath,line=yield tag tag=false if patter N in Line:target.send (Abspath) tag=true

Fifth stage: Print the file name that the line belongs to

@initdef printer (): While True:abspath=yield print (abspath) g = Search (Opener ("Error". Encode (' UT F-8 '), Printer ()))) G.send (R ' E:\Python\script\ function \test ')

Execution Result:

E:\Python\script\ function \test\aa\file1.txte:\python\script\ function \test\aa\bb2\file3.txt

This article is from the "Hyun-dimensional" blog, please be sure to keep this source http://xuanwei.blog.51cto.com/11489734/1953449

Python automated Transport Koriyuki advanced functions