Implementing an HTTP proxy using tornado

0x00 HTTP Proxy

HTTP proxy is a lot of use, there are public agents, but sometimes for work needs, such as analysis of application layer traffic, Do data access control, and even do monitoring and so on. Tornado provides some very convenient environments and APIs, and we can easily implement an HTTP proxy based on the tornado.

0X01 Implementation principle

The HTTP proxy is primarily a forwarding between the client and the Web server, a scenario that everyone is familiar with, but is limited to the HTTP protocol. For the case of HTTPS, this time the agent only as a TCP relay for information transfer, need to be processed separately.

0X02 Tornado implementation

Based on tornado can implement an asynchronous HTTP proxy, the performance is superior, the implementation is simple, the main use of the class is Asynchttpclient,iostream. Read the tornado source of the students may be not unfamiliar with these two classes. Here or simply, Asynchttpclient, as the name implies, is used to make asynchronous HTTP client requests, while iostream is a layer of encapsulation of the socket.
Asynchttpclient is used to handle the normal HTTP request, RequestHandler to obtain the client request, Proxy needs to resolve the client's request and use this class to request the server, get response, and then write to the client, finish the call.
For proxy as a TCP relay, you can actually use the original socket two head read and write data, but too much trouble. Tornado provides a iostream class, which can be seen as a socket wrapper class, which is much simpler to use than a socket, and 异步非阻塞 the socket is.
Talk is cheap, show me the code, do not say, look at the code, here for some reason, I can only post the key part of the code, hope to read this article of the students can write their own to use, in fact, it is not difficult.

Handling HTTP Requests

@tornado. Web.asynchronous     def get(self):        # GET request BodyBODY = Self.request.bodyif  notBody:body =None        Try:# Proxy Send requestRender_request (Self.request.uri, Callback=self.on_response, met                    Hod=self.request.method, Body=body, Headers=self.request.headers, follow_redirects=False, allow_nonstandard_methods=True)exceptTornado.httpclient.HTTPError asHttperror:ifHasattr (Httperror,' response ') andHttperror.response:self.on_response (Httperror.response)Else: Self.set_status ( -) Self.write (' Internal server error:\n '+ str (httperror)) Self.finish ()

Nothing to say, received a client request, go directly to request the server on the line. The asynchronous callback function is on_response, and this function handles the interaction between the proxy and the client, self.write(response.body) you know.
Here is a hole, that is, write headers, the response of the headers copy set again will be error, resulting in Access failure. My approach here is to write only self._headers the headers that exist in the RequestHandler.

TCP Relay Implementation

For 443-port or browser connect requests, the agent can only start from the TCP layer, forwarding the entire HTTP message, here is using the HTTP protocol in the Connect method, in the RequestHandler implementation of this method is OK.
By the way, when this method is called, the agent does not have to relate to the specific content requested by the HTTP layer, but forwards the message directly from the TCP layer to the server. When received, it is also forwarded to the client.

CONNECT www.web-tinker.com:80 HTTP/1.1Hostwww.web-tinker.com:80Proxy-ConnectionKeep-AliveProxy-AuthorizationBasic *Content-Length0

The specific implementation code is as follows:

    @tornado. Web.asynchronous     def connect(self):        ' for HTTPS connections, the proxy should act as a TCP relay '         def req_close(data):            ifConn_stream.closed ():return            Else: Conn_stream.write (data) def write_to_server(data):Conn_stream.write (data) def proxy_close(data):            ifReq_stream.closed ():return            Else: Req_stream.close (data) def write_to_client(data):Req_stream.write (data) def on_connect():            "' create callback for TCP relay 'Req_stream.read_until_close (Req_close, Write_to_server) conn_stream.read_until_close (Proxy_close, Write_to_clie NT) Req_stream.write (B ' http/1.0 Connection established\r\n\r\n ')Print ' starting Conntect to%s '% Self.request.uri# Get the socket for requestReq_stream = Self.request.connection.stream# Find the host port, typically 443Host, Port = (None,443) Netloc = Self.request.uri.split (': ')ifLen (netloc) = =2: host, Port = NetlocelifLen (netloc) = =1: host = netloc[0]# Create iostreams = socket.socket (socket.af_inet, socket. Sock_stream,0) Conn_stream = Tornado.iostream.IOStream (s) conn_stream.connect ((host, Port), On_connect)

I explain these two sentences:

req_stream.read_until_close(req_close, write_to_server)conn_stream.read_until_close(proxy_close, write_to_client)

Just two lines of code, plus 4 callback functions, completes the data transfer.
First, Req_stream is the socket between the proxy and the client, which can be obtained by httprequest the socket between the corresponding Iostream,proxy and the server is created by itself, here is conn_stream.
The Read_until_close method is provided in iostream to read the data until the socket is closed.
The function of the first line is to read the data from the socket between the client and proxy, read it, and write it to the socket between the proxy and the server, which is forwarded by proxy.
The second line is to write the server data to the client socket, as above, there is nothing to say, write the function is in four callback function.
Some wonder why Read_until_close has two callback functions, I understand that the first callback is called when it is closed, and the second callback is called when the data is read.
Write to use the effect is OK:

Copyright NOTICE: This article for Bo Master original article, without Bo Master permission not reproduced.

Implementing an HTTP proxy using tornado