Python coroutine and application (1): Introduction

Preface:I have been paying attention to how to make development easier, more efficient, and more error-prone. A similar effort has been made to edit a process based on SOA and workflow engine: http://blog.csdn.net/chgaowei/article/category/597.pdf. It has its own drawbacks: the coupling between scripts and programs is too large, and there are too many restrictions on scripts.

Later, I came into contact with and learned python, Which is concise and powerful. It should be a way of thinking to solve such problems. When I saw the yield keyword of the built-in Python generator that day, I felt that it was a breakthrough in solving the logic problem (which will be introduced later ). Later, I joined Python con2011 China. Many people mentioned coroutine and greenlet, especially the slogan "coroutine is the future. I was so excited that I had been thinking about a solution to the problem.

 What is coroutine?

For more information, see Wikipedia: http://zh.wikipedia.org/wiki/%e5%8d%8f%e7%a8%8b.

I have written an article about concurrent programming. I think this definition may be more appropriate: "A concurrency mechanism provided by programming languages ".

Let's take a look at the process and thread.

In the future, the computer will be able to process multiple tasks at the same time. The operating system has the concept of a process, and within the process, it can basically be considered that only one process is running in the current system, the operating system provides a very good encapsulation. The switching between processes is completed by the operating system.

One problem with a process is that switching between processes consumes a lot of computer resources, and the cost of applying for a new process is also very high. As a result, there was a thread, which generated much lower costs and the consumption of switching than the process, and made inter-thread communication very convenient.

Thread problems: 1) There is a thread security problem, and it is very difficult to locate the problem. 2) there is a limit on the number of threads in the process. 3) as the concurrency increases, the cost of thread generation and switching is also high.

Another solution to solve concurrency is Io multiplexing, which is indeed very efficient, but the Code complexity is also very high: it splits a process into nodes, scattered in multiple places, it is very unfavorable for development and maintenance (this is a frequently used solution ).

Okay. Let's see how coroutine solves these problems:

1) The generation cost of coroutine is lower. It is actually a piece of memory that records the stack information of previous calls. You can even further reduce the coroutine size by controlling the hierarchy of function calls. To generate a coroutine, you only need to apply for a piece of memory and assign a value.

2) switching is faster. It is basically the speed of memory copy.

3) No thread security issues. A process can have multiple coroutines at the same time, but only one coroutine is activated. In addition, the cooutine activation and sleep are controlled by programming, rather than by the kernel. This eliminates the thread security issues.

4) better readability. Compared with IO multiplexing, the service interface or I/O interface you call is asynchronous, but your code is fluent (in sequence) and is not disrupted by asynchronous and callback. The coroutine is asynchronous, But it encapsulates asynchronous events and callbacks to form a remote call interface.

Some materials:

On pycon 2011 China, Lai Yong-hao lists a node. js code that is implemented asynchronously and the python coroutine-style code that can explain the problem: http://www.slideshare.net/laiyonghao/python-webgame-10452102.

Lai Yonghao's article on coroutine is very good and can be used as a reference. Unfortunately, it was not completed: one of the three articles on coroutine (first contact with coroutine ).

Lai Yonghao also made an opportunity for Google protocol buffer and greenlet's remote call framework Abu. rpc. If you are interested, please refer to it (you haven't made any time to read it before writing this article ).

Programming Languages that provide coroutine mechanisms:

Currently, many languages provide coroutine mechanisms or use third-party modules to implement coroutine. Currently, I know that languages supporting coroutine include python, Lua, Erlang, go, Io, Ruby, and C. I even see some articles that use go and switch or longjmp/setjmp in C to implement coroutine, but it is difficult to use it in actual projects.

 Python coroutine implementation:

Yield can implement coroutine. In addition, there are many third-party versions, such as greenlet.

 What can coroutine do?

1) description logic: I mainly use coroutine to describe the logic. A process may need to call multiple interfaces, many of which are asynchronous. It is difficult to describe this. Some problems can be solved using threads, but the complexity is improved.

2) Improve concurrency: Mainly used in io-intensive applications. Gevent is a framework for processing concurrency Based on greenlet. The difference is that the events and interfaces here are Io interfaces.

 Defects:

Multi-core cannot be used. However, it can be solved through process + coroutine.