Python network library gevent based on the association process

Importdef  test1 ():    print     gevent.sleep (0)      Printdef  test2 ():    print     gevent.sleep (0)      Print  gevent.joinall ([    gevent.spawn (test1),    gevent.spawn (test2),])

Explanation, the "gevent.spawn ()" method creates a new Greenlet-the-thread object and runs it. The "Gevent.joinall ()" Method waits for all incoming Greenlet to finish running before exiting, and this method can accept a "timeout" parameter to set the time-out, in seconds. Run the above program and execute the following sequence:

1. Enter the Test1 first, print 12
2. When encountering "Gevent.sleep (0)", Test1 is blocked, automatically switches to the co-test2, prints 56
3. After the test2 is blocked, the test1 block is closed and automatically switches back to test1, printing 34
4. When Test1 is finished running, the Test2 block is closed, and then automatically switches back to test2, printing 78
5. The program exits after all the processes have been executed

So the output that the program runs down is:

12563478

Greenlet after a process has finished running, you must explicitly toggle it, or it will return its parent co-process. In Gevent, when a process is finished running, it automatically dispatches those unfinished processes.

 import   gevent  import   socket urls  = [ " www.baidu.com   ", "  www.gevent.org   ", "   www.python.org   " ]jobs  = [ Gevent.spawn (socket.gethostbyname, url) for  URL  In   Urls]gevent.joinall (jobs, timeout  =5 print  [job.value for  job in  Jobs] 

We obtain the IP address of three websites by the association process separately, because the open remote address can cause IO blocking, so the gevent will automatically dispatch different co-processes. In addition, we can get the return value of the co-function by the "value" property of the co-process object.

Monkey Patch Monkey Patching

In fact, the above program to run the same time without the co-process is the same, is the sum of three web site open time. But in theory, the process is non-blocking, the running time should be equal to the longest site open time ah? In fact, this is because the Python standard database socket is blocking, DNS parsing can not be concurrent, including the same as the Urllib library, so in this case the use of the process is completely meaningless. So what?

One way is to use the socket module under Gevent, which we can import through the "from gevent import socket". But the more common approach is to use monkey pudding (Monkey patching):

 fromGeventImportmonkey; Monkey.patch_socket ()ImportgeventImportSocket URLs= ['www.baidu.com','www.gevent.org','www.python.org']jobs= [Gevent.spawn (socket.gethostbyname, URL) forUrlinchUrls]gevent.joinall (jobs, timeout=5) Print[Job.value forJobinchJobs

The first line of the above code is to the socket standard library Monkey patch, after the socket standard library classes and methods will be replaced with non-blocking, all the other code is not modified, so that the efficiency of the process is really reflected. Other standard libraries in Python also have blocking situations, and Gevent provides a "Monkey.patch_all ()" method to replace all standard libraries.

 from import Monkey; Monkey.patch_all ()

The use of monkey patches mixed, but the official website is still recommended to use "Patch_all ()", and in the first line of the program to execute.

Get the process status

The process state has been started and stopped, and can be judged by the "started" property of the co-object and the "Ready ()" method, respectively. For a stopped association, you can use the "successful ()" method to determine whether it runs successfully and does not throw an exception. If the process finishes executing with a return value, it can be obtained through the "value" property. In addition, the exception that occurs during the Greenlet process is not thrown out of the coprocessor, so you need to use the "exception" property of the Coprocessor object to get the exception in the thread. The following example illustrates the use of various methods and properties.

#Coding:utf8Importgeventdefwin ():return 'You win!' deffail ():RaiseException ('You failed!') Winner=Gevent.spawn (Win) loser=gevent.spawn (fail)Printwinner.started#TruePrintloser.started#True #The exception that occurs in Greenlet is not thrown out of the greenlet. #the console will play StackTrace, but the program will not stopTry: Gevent.joinall ([winner, loser])exceptException as E:#This section will never be executed .    Print 'This would never be reached' PrintWinner.ready ()#TruePrintLoser.ready ()#True PrintWinner.value#' You win! 'PrintLoser.value#None PrintWinner.successful ()#TruePrintLoser.successful ()#False #This can be done by raise loser.exception or Loser.get ()#to throw an exception in the associationPrintLoser.exception

Run timeout for the co-process

You can pass the timeout parameter in the "Gevent.joinall ()" method to set the timeout, and we can set the time-out in the global scope:

Import gevent  from Import  = Timeout (2)  #  2 secondsdef  Wait ():    gevent.sleep ( Tentry:    gevent.spawn (Wait). Join ()except  Timeout:     Print('Couldnot complete')

In the example above, we set the time-out to 2 seconds, after which all the threads run, and a "timeout" exception is thrown if more than two seconds. We can also set the timeout within the WITH statement so that the setting is valid only in the WITH statement block:

With Timeout (1):    gevent.sleep (10)

In addition, we can specify the exception thrown by the timeout to replace the default "timeout" exception. For example, the timeout in the following example throws our custom "Toolong" exception.

class Toolong (Exception):     Pass With Timeout (1, Toolong):    gevent.sleep (10)

Communication between co-processes

An event object can be used for asynchronous communication between Greenlet. The "Wait ()" method of the object can block the current coprocessor, and the "set ()" Method can wake up the previously blocked threads. In the example below, 5 waiter will wait for the event evt, and when the setter Association sets the evt event after 3 seconds, all waiter threads are awakened.

#Coding:utf8Importgevent fromGevent.eventImportEvent evt=Event ()defsetter ():Print 'Wait for me'Gevent.sleep (3)#3 seconds to wake up all the threads waiting on the evt    Print "Ok, I ' m done"Evt.set ()#Wake up defWaiter ():Print "I ' ll wait for you"evt.wait ()#wait    Print 'Finish Waiting'Gevent.joinall ([Gevent.spawn (Setter), Gevent.spawn (Waiter), Gevent.spawn (Waiter), Gevent.spawn (waiter), Gevent.spawn (Waiter), Gevent.spawn (waiter)])

In addition to the event events, Gevent also provides a asyncresult event that can pass messages on wakeup. Let's make the following changes to the setter and waiter in the example above:

 fromGevent.eventImportasyncresultaevt=AsyncResult ()defsetter ():Print 'Wait for me'Gevent.sleep (3)#3 seconds to wake up all the threads waiting on the evt    Print "Ok, I ' m done"Aevt.set ('hello!')#wake up and deliver the message defWaiter ():Print("I ' ll wait for you") Message= Aevt.get ()#wait, and get the message when it wakes up    Print 'Got Wake up message:%s'% message

Queues Queue

The Gevent queue object allows secure access between the Greenlet threads. Run the following program and you will see that 3 consumers will consume the product in the queue separately, and the consumed product will not be picked up by another consumer:

Importgevent fromGevent.queueImportQueue Products=Queue ()defConsumer (name): while  notproducts.empty ():Print '%s got product%s'%(name, Products.get ()) gevent.sleep (0)Print '%s Quit' defproducer (): forIinchXrange (1, 10): Products.put (i) Gevent.joinall ([Gevent.spawn (producer), Gevent.spawn (consumer,'Steve'), Gevent.spawn (consumer,'John'), Gevent.spawn (consumer,'Nancy'),])

Both the put and get methods are blocking, both of which have non-blocking versions: Put_nowait and get_nowait. If the queue is empty when the Get method is called, an "Gevent.queue.Empty" exception is thrown. ‘

Signal Volume

The semaphore can be used to limit the number of concurrent threads. It has two methods, acquire and release. As the name implies, acquire is the acquisition of the semaphore, and release is released. When all semaphores have been acquired, the remaining coprocessor can only wait for any one of the co-processes to release the semaphore before it can be run:

Importgevent fromGevent.corosImportboundedsemaphore sem= Boundedsemaphore (2) defworker (N): Sem.acquire ()Print('Worker%i acquired Semaphore'%N) gevent.sleep (0) sem.release ()Print('Worker%i released Semaphore'%N) Gevent.joinall ([Gevent.spawn (Worker, I) forIinchXrange (0, 6)])

In the example above, we initialized the "Boundedsemaphore" semaphore and set the number to 2. So at the same time, only two worker threads can be dispatched. After the program runs, the results are as follows:

1123234455 released Semaphore

Co-local variables

Similar to threads, the co-process also has local variables, which are variables that are accessible only within the current process:

Importgevent fromGevent.localImportLocal Data=Local ()defF1 (): data.x= 1Printdata.xdefF2 ():Try:        Printdata.xexceptAttributeerror:Print 'x is not visible'Gevent.joinall ([Gevent.spawn (F1), Gevent.spawn (F2)])

By storing the variable in the local object, you can limit its scope within the current coprocessor and throw an exception when other threads are accessing the variable. The local variables with the same name can be found between the different processes, and do not affect each other. Because the implementation of the co-local variable, it is stored in the "greenlet.getcurrent ()" returned as the key value in the private namespace.

Practical application

Based on flask chat room

https://github.com/sdiehl/minichat/blob/master/app.py

Python network library gevent based on the association process