02-twisted long link problem for Building Web server socket | 07. Miscellaneous | Python

02-twisted the problem of constructing the socket persistent link of Web Server

Zheng Zhengyi 201005 is affiliated with 07. Miscellaneous

Background

Easy to useCodeYou can build a web server:

From twisted. Internet import Reactor
From twisted. Web. Server Import site
From my_webhandler import *
Reactor. listentcp (8080, site (mywebresource. Setup ()))

For more complex application, see the IBM Document Library: Part 1 describes Asynchronous Server programming; Part 2 describes advanced Web Service programming; Part 2 describes dynamic Web servers using woven templates; section 4th describes how to use SSH. Or using ing and using the twisted. Web server.

 

Sometimes, too many socket connections are opened.

When this web server is used to receive various requests from the pubsubhubbub hub server, a major problem occurs:

Over time, more and more socket connections in the established status gradually reached more than 500,

Tcp x. x. x. x: 8080 72.14.192.68: 55693 established
TCP x. x: 8080 74.125.126.80: 59064 established

Eventually, the service becomes abnormal."Too plugin file descriptors in select", When this exception occurs, it cannot be saved and the service can only be restarted.

Here, the knowledge point is that in windows, the select module file descriptor is a maximum of 512, and in Linux, the limit is 32767. If the limit is exceeded, an exception similar to the above occurs.

 

For the moment, whether this problem involves the habit of pubsubhubbub hub maintaining a persistent link (hubs may leave their TCP connections open and reuse them to make HTTP requests for freshly published events) and maybe reusing connections.

Since the server (Our webserver) cannot guarantee that idle connections are always disconnected, you can set idle timeout to solve this problem.

 

Timeout settings of twisted. Web. server. Site

The initialization function of the twisted. Web. server. site class has an optional parameter timeout. Its default value is 60*60*12, which should be 12 hours.
The timeout value is assigned to the timeout attribute of twisted. Internet. Protocol through the initialization function of twisted. Web. http. httpfactory, so that it can be found in the underlying HTTP protocol.Idle connection timeoutThen it is processed by timeoutmixin.

12 hours is too long. Therefore, many socket connections in the established State will accumulate. So we shortened it to 15 minutes. So, we only need to start executing:
Reactor. listentcp (8080, site (mywebresource. Setup (), Timeout = 60*15))
You can.

In this way, when the socketAfter receiving data(In this case, the self. resettimeout () reset is called.) When the idle connection time times out, the twisted. Protocols. Policies. timeoutmixin. timeoutconnection function is called by default. Its definition is:

Def timeoutconnection (Self ):
"" Called when the connection times out.
Override to define behavior other than dropping the connection.
"""
Self. Transport. loseconnection ()

That is, close the connection. In this case, a prompt is displayed:

Timing out client: ipv4address (TCP, '72. 14.192.68 ', 43949)

After practice, the socket connections occupied by the Web server can be greatly reduced.

 

What is timeoutmixin?

Someone complained about twisted like me:
"Client will not close the connect unit it disconnects itself. but the server can't control the connect. in some critical environment, the server has to maintain a lot of Connect which maybe is idle. 』
Someone replied that twisted. Protocols. Policies. timeoutmixin can allow the server to actively disconnect:
Class timeouttester (protocol. Protocol, policies. timeoutmixin ):
Timeout = 3
Timedout = 0

Def connectionmade (Self ):
Self. setTimeout (self. Timeout)

Def datareceived (self, data ):
Self. resettimeout ()
Protocol. Protocol. datareceived (self, data)

Def connectionlost (self, reason = none ):
Self. setTimeout (none)

Def timeoutconnection (Self ):
Self. timedout = 1
For another example, see:
Http://code.google.com/p/proxy65/source/browse/trunk/proxy65/socks5.py

 

Reference resources:

1. How to Set Client timeout with reactor. listentcp?

2. Refer to the twisted document knowing when we're not wanted:

 
『
 

Sometimes it is useful to know when the other side has broken the connection. Here is an example which does that:

From twisted. Web. Resource import Resource
From twisted. Web import server
From twisted. Internet import Reactor
From twisted. Python. util import println

Class exampleresource (Resource ):

Def render_get (self, request ):
Request. Write ("Hello World ")
D = request. policyfinish ()
D. addcallback (lambda _: println ("finished normally "))
D. adderrback (println, "error ")
Reactor. calllater (10, request. Finish)
Return server. not_done_yet

Resource = exampleresource ()
 

This will allow us to run statistics on the log-file to see how many users are frustrated after merely 10 seconds.

 
』
 

3. Problems with Windows ports frequently opened and closed;

4. Choosing a TCP port for a network service;

5. 02-twisted: long socket connection problem for Web Server construction | 07. Miscellaneous | Python;

6. problems that may occur when Windows frequently opens or closes ports | 07. Miscellaneous.