Python encapsulation iocp

: Source code download: http://blog.csdn.net/jgood/article/details/6626872

Python is a concise and elegant language. Her high productivity allows us to solve our immediate problems with minimal manpower, material resources, time, and financial cost. But when the current problems are solved, some long-term problems have to be considered.

Using the socket module provided by python to write network server programs makes it difficult to withstand thousands of connections. I used to refer to the netstream module learned by LI Yong haotong and used the socket module provided by python to compile the communication model. However, the actual effect is not very good. When the number of connections reaches about 800, the CPU usage reaches 70%-80%. Of course, it may be that the quality of the code I write is poor. Darkspy is a master, and its own communication model can achieve very high efficiency, but I am far from reaching his level. According to Dr. Li, the best way to use iocp is to create a high-performance network communication platform on Windows. As a result, I have been trying to use python to encapsulate iocp over the past few days.

Python itself is implemented in C language and supports the embedded extension of C/C ++ very well. The C/C ++ code can be called by python as long as it is simply encapsulated, and vice versa. Therefore, using C/C ++ to encapsulate iocp and then using python for calling should be a good solution.

There are a lot of information about iocp on the Internet, and the ready-made code is also everywhere. Although it can only be implemented using two or three windows API functions, it still takes a general discount to understand iocp and the overlapping Io model of windows. I spent a lot of time learning the overlapping I/O model of windows. For some examples on the internet, I wrote the simplest example using C ++ code and encapsulated it using python. I still have a vague understanding of the overlapping I/O models and iocp for windows, so I should digest them in the next period.

Use iocp to connect/disconnect the underlying client socket and send and receive data. Send the data received by iocp to python for processing. After processing is completed, the data is sent back to the client through iocp. The general process is as follows:

In my encapsulated iocp module, only five methods are provided:

Pyiocp. Start (port, max_conn_count, buffer_size)

This method is used to start a listener. The parameter port is the listening port number; max_conn_count indicates the maximum number of connections allowed; buffer_size indicates the buffer size for Low-layer data transmission and receiving. If the function is successfully started, 0 is returned.

Pyiocp. Stop ()

The stop method is used to disable the listener.

Pyiocp. Recv ()

The Recv () method returns the received data. The data here is not only the data sent/received by the socket, but also provided to python in the form of data for connection creation and disconnection. The data type can be determined based on the first element of the returned tuples. For example, if the user connects to the server, a triple (1, socket_id, "IP: Port") is returned by calling Recv (); when the user disconnects, a triple (2, socket_id, ""); when receiving data, return (4, socket_id, 'Here is data'); if no data exists, call the Recv () method will return (0, 0 xffffffff, "").

Pyiocp. Send (socket_id, data)

The send () method is used to send data. If the call is successful, the length of the sent data is returned. If the operation fails, a value smaller than 0 is returned.

Pyiocp. Close (socket_id)

The close () method is used to close a specified connection.

I wrote a simple echo server using the iocp encapsulation module to test the performance. When 4000 clients are connected and a large amount of data is communicated, the system consumes about 20% of the resources. Although there is still a gap between this and iocp performance indicators on the Internet.

Network Communication is no longer a performance bottleneck for python servers. This is what I want to achieve. Xi ~~ I am very grateful to darkspy and LAI Yonghao for their suggestions and help.

 

Source code download is not provided for the time being. If you need it, you can leave a message and leave an email.