Python's own HTTP module

quite a long time no blog, because the blogger began this year another good internship experience, study plus project, Time is full, thank you for this year's two experiences, let me touch the Golang and Python, learning different languages, can jump out of the C + + thinking limits, Learn Golang and Python's excellent features as well as understanding the different languages in various scenarios, and before learning Linux and C + +, it also enabled me to quickly get started with Golang and Python;

I study the habit, in addition to learning how to use, but also like to study the source code, learning operation mechanism, so that it will be handy or say, use these languages or frameworks, and usually eat to sleep the same, very natural; because recently have access to the bottle and Flask Web framework, so want to see the two source code, But these two frameworks are based on Python's own HTTP, so there's this article;

Python http Simple example

Python HTTP framework mainly consists of server and handler, the server is mainly used to establish network model, such as the use of epoll monitoring Socket;handler to handle each ready socket; look at Python first. HTTP simple to use:

Import sysfrom http.server Import httpserver,simplehttprequesthandlerserverclass = Httpserverhandlerclass = simplehttprequesthandlerif__name__ = = ' __main__ ': port = Int (sys.argv[2]) server_address = (sys.argv[1],port) httpd = ServerClass (Server_address,handlerclass) sa=httpd.socket.getsockname () print ("Serving HTTP on", Sa[0], "Port", sa[1], "...") try:httpd.serve_forever () except Keyboardinterrupt:print ("\nkeyboard interrupt received, exiting.") Httpd.server_close () sys.exit (0)

To run the above example, you can get the following:

Python3 myhttp.py 127.0.0.1 9999

At this point, if you create a new index.html file in the current folder, you can access the Index.html page through http://127.0.0.1:9999/index.html.

The server class for this example uses the Httpserver,handler class to be simplehttprequesthandler, so when Httpserver hears that a request arrives, Just throw this request to the Simplehttprequesthandler class for processing; OK, after that, we begin to analyze the server and handler separately.

HTTP Server

The HTTP module is designed to take full advantage of object-oriented inheritance polymorphism, since there is code for the TFS file system, so there is less pressure to look at Python http, and the inheritance of the server first

+------------------++------------+| TCPServer Base Class | | Baseserver +-------->| Turn on event loop monitoring |+-----+------+ | Processing Client Requests | | +------------------+ V +-----------------+ +------------+| Httpserver Base Class | | TCPServer +-------->+ settings Monitor Socket |+-----+------+ | Open Listening | | +-----------------+ v+------------+| Httpserver | +------------+

The inheritance relationship as shown, wherein Baseserver and tcpserver in the file socketserver.py,httpserver in http/server.py; let's first look at Baseserver;

Baseserver

Because Baseserver is the base class for all servers, baseserver as much as possible to abstract all the common features of the server, such as the turn on the event listener Loop, which is the commonality of each server, so this is also baseserver to make; Let's take a look at the main code section of Baseserver

Defserve_forever (self, poll_interval=0.5): Self.__is_shut_down.clear () Try:with_serverselector () Asselector: Selector.register (self, selectors. Event_read) Whilenotself.__shutdown_request:ready = Selector.select (poll_interval) ifready:self._handle_request_ Noblock () self.service_actions () Finally:self.__shutdown_request = False Self.__is_shut_down.set ()

The selector in the code actually encapsulates the IO multiplexing of the select,poll,epoll, and then registers the socket that the service listens on to Io multiplexing, turns on event snooping, and when there is a client connection, this calls Self._handle_ Request_noblock () to process the request; next look at what this handler does;

Def_handle_request_noblock (self): try:request, client_address = Self.get_request () Exceptoserror: Returnifself.verify_request (Request, client_address): try:self.process_request (Request, client_address) except: Self.handle_error (Request, client_address) self.shutdown_request (Request) Else:self.shutdown_request (Request)

The _handle_request_noblock function is an intrinsic function that first receives a client connection request, which in fact encapsulates the system call accept function, validates the request, and finally calls Process_request to process the request; get_ Request is a subclass method because TCP and UDP receive client requests are not the same (TCP has a connection, UDP is not connected)

Let's look at what the Process_request has done next.

Defprocess_request (self, request, client_address): self.finish_request (Request, client_address) Self.shutdown_ Request #-------------------------------------------------Deffinish_request (self, request, Client_ Address): Self. Requesthandlerclass (Request, client_address, self) defshutdown_request (self, request): Self.close_request (Request)

The Process_request function first calls the finish_request to process a connection, and after the processing is finished, calls the Shutdown_request function to close the connection; finish_ The request function internally instantiates a handler class, and the client socket and address passed in, indicating that the handler class at the end of the initialization, the completion of the processing of requests, the subsequent analysis of the handler when the next look;

The above is what baseserver do, this baseserver can not be used directly, because some functions have not been implemented, just as the abstraction layer of TCP/UDP;

First call Serve_forever Open event monitoring;
Then, when a client request arrives, the request is submitted to handler for processing;

TCPServer

By the above Baseserver abstraction function, we can know TCPServer or udpserver should complete the function has, initialize the listening socket, and binding monitoring, and finally when there is a client request, receive this client; let's see the code.

Baseserver==>def__init__ (self, server_address, Requesthandlerclass): "" "Constructor. May is extended, do not override. "" "self.server_address = server_address self. Requesthandlerclass = Requesthandlerclass Self.__is_shut_down = Threading. Event () Self.__shutdown_request = false#------------------------------------------------------------------------- -------tcpserver==>def__init__ (self, server_address, Requesthandlerclass, bind_and_activate=true): BaseServer._ _init__ (self, server_address, requesthandlerclass) Self.socket = Socket.socket (self.address_family, Self.socket_type ) Ifbind_and_activate:try:self.server_bind () self.server_activate () Except:self.server_close () Raise

TCPServer initializes the initialization function of the base class Baseserver, initializes the server address, handler class, and so on, initializes its own listener socket, and finally calls the Server_bind bound socket, server_activate listening socket

Defserver_bind (self): ifself.allow_reuse_address:self.socket.setsockopt (socket. Sol_socket, SOCKET. SO_REUSEADDR, 1) self.socket.bind (self.server_address) self.server_address = Self.socket.getsockname () defserver_ Activate (self): Self.socket.listen (self.request_queue_size)

TCPServer also implements another function, which is to receive client requests,

Defget_request (self): returnself.socket.accept ()

If you have learned Linux programming before, then look at the code should be very familiar with, because the function name and Linux provided by the system call name is identical, there is not much to say;

TCPServer in fact already put TCP-based server main frame, so httpserver on the basis of inheritance tcpserver, just overloaded server_bind function, set reuse_address, etc.

OK, here to analyze the opening process of the above example program;

httpd = ServerClass (server_address,handlerclass) This line of code in the initialization of Httpserver, is mainly called the base class TCPServer initialization method, the initialization of the listening socket, and binding and listening;
Httpd.serve_forever () This line of code calls the base class Baseserver Serve_forever method, turn on the listening loop, waiting for the client connection;
If you have seen Redis or some background components of the source code, this concurrency model should be familiar with; OK, after analyzing the server, let's look at how handler handles client requests.

Handler of HTTP

The handler class mainly analyzes the TCP layer's handler and HTTP application Layer HANDLER,TCP layer handler is not used, because the TCP layer is only responsible for transmitting bytes, but do not know how to parse the bytes received, how to handle the Therefore, if the application layer protocol is to use the TCP protocol, it must inherit the TCP handler and then implement the handle function, for example, the handler of the HTTP layer implements the handle function, resolves the HTTP protocol, processes the business request, and returns the result to the client. First, take a look at the TCP layer's handler

TCP Layer Handler

TCP layer handler mainly have Baserequesthandler and Streamrequesthandler (both in socketserver.py file), first look at the Baserequesthandler code,

Classbaserequesthandler:def__init__ (self, request, client_address, server): Self.request = Request Self.client_ Address = client_address Self.server = Server Self.setup () try:self.handle () Finally:self.finish () Defsetup (self): Passdefhandle (self):p assdeffinish:p

Before looking at the server, knowing that processing a client request is done in the initialization function of the handler class, and that the base class initializes the function, we know that processing the request goes through three processes:

1. Setup makes some settings for the client socket;

2. Handle the function that really handles the request;

3. Finish closes the socket read and write request;
   

This baserequesthandler is the handler top level base class, just abstract out handler overall framework, and no actual processing; we look at the TCP handler,

Classstreamrequesthandler (baserequesthandler): timeout = None Disable_nagle_algorithm = Falsedefsetup (self): Self.connection = Self.requestifself.timeoutisnotNone:self.connection.settimeout (self.timeout) Ifself.disable_ Nagle_algorithm:self.connection.setsockopt (socket. IPPROTO_TCP, Socket. Tcp_nodelay, True) self.rfile = Self.connection.makefile (' RB ', self.rbufsize) self.wfile = Self.connection.makefile (' WB ', self.wbufsize) deffinish (self): Ifnotself.wfile.closed:try:self.wfile.flush () Exceptsocket.error:pass Self.wfile.close () Self.rfile.close ()

TCP handler implements the setup and finish functions, the Setup function sets the timeout time, the Nagle algorithm is opened, and the socket read/write cache is set; the finish function closes the socket read and write;

From the above two TCP layer handler know, to implement an HTTP-based server handler, only need to inherit Streamrequesthandler class, and implement handle function, so this is the HTTP layer handler mainly do;

HTTP Layer Handler

From the previous TCP layer handler introduction, we know that the HTTP layer handler in the inheritance of TCP layer handler based on the main implementation of the handle function to handle the client's request, or directly read the code it;

Defhandle (self): self.close_connection = True self.handle_one_request () whilenotself.close_connection:self.handle_ One_request ()

This is the handle function of Basehttprequesthandler, where the handle function invokes the Handle_one_request function to process a request, and by default it is a short link, so after a request is executed, Does not enter the while loop to process the next request on the same connection, but inside the handle_one_request function, if the connection in the request header is Keep_ Alive or HTTP version is greater than or equal to 1.1, you can maintain long links, and then see how the Handle_one_request function is handled;

Defhandle_one_request (self): Try:self.raw_requestline =self.rfile.readline (65537) Iflen (self.raw_requestline) > 65536:self.requestline = ' Self.request_version = ' Self.command = ' Self.send_error (httpstatus.request_uri_too_long ) Returnifnotself.raw_requestline:self.close_connection = Truereturnifnotself.parse_request (): Return mname = ' Do_ ' + Self.commandifnothasattr (self, Mname): Self.send_error (httpstatus.not_implemented, "unsupported Method (%r)"% Self.command) Return method = GetAttr (self, Mname) method () Self.wfile.flush () except Socket.timeout as E:self.log_error ( "Request timed out:%r", e) self.close_connection = Truereturn

The handle_one_request execution process is as follows:

1. First, call Parse_request to parse the client HTTP request content

2. The function method of the request is constructed through the "Do_" +command

3. Call the method function, process the business and return the response to the client
   

This basehttprequesthandler is an HTTP handler base class and therefore cannot be used directly because it does not define a request handler function, which is the method function Fortunately, Python provides us with a simple simplehttprequesthandler, which inherits the Basehttprequesthandler and implements the request function; Let's look at the Get function:

# simplehttprequesthandler#---------------------------------------------defdo_get (self): "" "Serve a GET request." " f = Self.send_head () iff:try:self.copyfile (f, Self.wfile) Finally:f.close ()

This get function first calls the Do_get function to return the response header to the client, returns the requested file, and finally calls the CopyFile function to return the request file to the client via the connection;

Above is the HTTP module most basic content, finally, summarizes the following example program handler part:

1. The server passes the request to the Simplehttprequesthandler initialization function;

2. Simplehttprequesthandler in the initialization section, the client connection some settings;

3. Then call the handle function to process the request;

4. The handle function then calls Handle_one_request to process the request;

5. Inside the Handle_one_request function, parse the request and find the request handler function;

6. My previous access is a get access, so call the Do_get function directly to return the index.html file to the client;
   

The Python HTTP module has been analyzed to the end; it is not known that Python's own HTTP module is not very convenient to use because it calls the request function through the request method, so that when the same method is called very many times, such as the Get and post methods, This can cause this request function to be unusually large, the code is poorly written, and the various circumstances are judged; Of course simplehttprequesthandler is just a simple example provided by Python;

Of course, Python officially provides a better framework for HTTP, Wsgi server and WSGI application, and the next article analyzes the Wsgiref module and bottle from Python, and then analyzes flask;