Python with its own HTTP module detailed _python

Quite a long time did not write a blog, because the blogger began this year another good internship experience, learn to add projects, time is full; Thank you for this year's two experiences, let me contact with Golang and python, learning different languages, can jump out of the previous learning C + + thinking restrictions, Learn the good features of Golang and Python and learn how to use different languages in different scenarios, and before learning Linux and C + +, I quickly started Golang and python;

My habit of learning, in addition to learning how to use, but also like to study the source code, learning to run the mechanism, so that use it will be handy or say, use these languages or frameworks, and usually eat and sleep like, very natural; because recently have contacted the bottle and Flask Web framework, so want to look at these two source, But these two frameworks are based on Python's own HTTP, so there's this article;

Python http Simple example

The Python HTTP framework consists primarily of server and handler, and server is primarily used to build network models, such as the use of Epoll listening Socket;handler to handle each ready socket; Let's take a look at Python. HTTP simple to use:

Import SYS from
http.server import httpserver,simplehttprequesthandler

serverclass = Httpserver
Handlerclass = Simplehttprequesthandler

if__name__ = = ' __main__ ':
 port = Int (sys.argv[2))
 server_address = (sys.argv[1],port)
 httpd = ServerClass (server_address,handlerclass)

sa=httpd.socket.getsockname ()
Print ("Serving HTTP on", Sa[0], "Port", sa[1], "...")

try:
 httpd.serve_forever ()
 except Keyboardinterrupt:
Print ("\nkeyboard interrupt received, exiting.")
 Httpd.server_close ()
 sys.exit (0)

To run the above example, you can get the following:

Python3 myhttp.py 127.0.0.1 9999

At this point, if you create a new index.html file in the current folder, you can access the Index.html page through http://127.0.0.1:9999/index.html.

The server class for this example uses the Httpserver,handler class to be simplehttprequesthandler, so when Httpserver hears that a request arrives, Leave this request to the Simplehttprequesthandler class to handle; OK, after we get to this, we start to analyze the server and handler separately.

HTTP Server

The HTTP module was designed to take advantage of object-oriented inheritance polymorphism, since there were previous code that looked at the TFS file system, so it was less stressful to look at Python http; first give the server's inheritance relationship

 +------------------+
+------------+| tcpserver base class |
| Baseserver +-------->| Turn on event loop monitoring |
+-----+------+ | Processing client requests |
 | +------------------+
 v +-----------------+
+------------+| httpserver base class |
| TCPServer +-------->+ set listening socket |
+-----+------+ | Open Monitor |
 | +-----------------+
 v
+------------+
| Httpserver | 
+------------+

The inheritance relationship is shown in the image above, where Baseserver and TCPServer in the file Socketserver.py,httpserver in http/server.py; first we look down baseserver;

Baseserver

Because Baseserver is the base class for all servers, Baseserver can abstract all server commonalities as much as possible, such as opening the event listener loop, which is the commonality of each server, so that's what baseserver does; Let's take a look at the main code section of Baseserver

Defserve_forever (self, poll_interval=0.5):
 self.__is_shut_down.clear ()
try:
with_serverselector () Asselector:
 selector.register (self, selectors. Event_read)

whilenotself.__shutdown_request:
 ready = Selector.select (poll_interval)
Ifready:
 Self._handle_request_noblock ()

 self.service_actions ()
finally:
 self.__shutdown_request = False
 Self.__is_shut_down.set ()

The selector in the code is actually encapsulates the IO multiplexing of the select,poll,epoll, and then registers the socket with the service itself to IO Multiplexing, opens the event listener, and when there is a client connection, the call Self._handle_ Request_noblock () to process the request, and then look at what the handler does;

Def_handle_request_noblock (self):
try:
 request, client_address = Self.get_request ()
exceptoserror:
return
ifself.verify_request (Request, client_address):
try:
 self.process_request (Request, client_address)
except:
 self.handle_error (Request, client_address)
 self.shutdown_request (Request
Else:
 self.shutdown_request (Request)

The _handle_request_noblock function is an internal function that receives the client connection request first, and the underlying is actually encapsulating the system call accept function, then validating the request, and finally invoking the process_request to process the request; get_ Request is a subclass of methods because TCP and UDP receive client requests are not the same (TCP is connected, UDP is not connected)

Let's take a look at what process_request specifically did;

Defprocess_request (self, request, client_address):
 self.finish_request (Request, client_address)
 Self.shutdown_request (Request)
#-------------------------------------------------
deffinish_request ( Self, request, client_address):
 self. Requesthandlerclass (Request, client_address, self)

defshutdown_request (self, request):
 self.close_request (Request)

The Process_request function calls the finish_request to process a connection, and after the process is finished, the Shutdown_request function is invoked to close the connection; Request function internal instantiation of a handler class, and the client socket and address passed in, indicating that the handler class at the end of the completion of the request processing, this and other follow-up analysis handler;

The above is what baseserver do, this baseserver can not be used directly, because some functions have not yet been implemented, but as a tcp/udp abstraction layer;

First, call Serve_forever to open the event to listen;
Then when a client request arrives, the request is handed over to handler for processing;

TCPServer

The function abstracted by the above baseserver, we can know TCPServer or udpserver should complete the function has, the initialization listens socket, and binds listens, finally when has the client request, receives this client; Let's take a look at the code.

Baseserver==>
def__init__ (self, server_address, Requesthandlerclass): "" "
constructor. May is extended, does not override.
 "" " self.server_address = Server_address
 self. Requesthandlerclass = requesthandlerclass
 Self.__is_shut_down = Threading. Event ()
 self.__shutdown_request = False
#--------------------------------------------------------------- -----------------
tcpserver==>
def__init__ (self, server_address, Requesthandlerclass, Bind_and_ Activate=true):
 baseserver.__init__ (self, server_address, requesthandlerclass)
 Self.socket = Socket.socket (self.address_family,
 self.socket_type)
ifbind_and_activate:
try:
 self.server _bind ()
 self.server_activate ()
except:
 self.server_close ()
raise

TCPServer initializes the initialization function of the base class Baseserver, initializes the server address, handler class, and so on, initializes its own listening socket, and finally invokes the server_bind binding socket, server_activate listening socket

Defserver_bind (self):
ifself.allow_reuse_address:
 self.socket.setsockopt (socket. Sol_socket, SOCKET. SO_REUSEADDR, 1)
 self.socket.bind (self.server_address)
 self.server_address = Self.socket.getsockname ()

defserver_activate (self):
 Self.socket.listen (self.request_queue_size)

TCPServer also implements another function, which is to receive client requests,

Defget_request (self):
returnself.socket.accept ()

If you've ever learned Linux programming before, look at the code as familiar, because the name of the function is exactly the same as the system call given by Linux, and it's not much to say here;

TCPServer has actually put up the server main frame based on TCP, so httpserver on the basis of inheriting tcpserver, just overload the Server_bind function, set reuse_address, etc.

OK, here is an analysis of the above example program opening process;

httpd = ServerClass (server_address,handlerclass) This line of code when initializing Httpserver, primarily calls the base class TCPServer initialization method, initializes the listening socket and binds and listens;
Httpd.serve_forever () This line of code calls the base class Baseserver Serve_forever method, open the listening loop, waiting for the client connection;
If you have seen Redis or some background components of the source code, this concurrency model should be very familiar with; OK, after analyzing the server, next look at how handler handles client requests.

Handler of HTTP

The handler class mainly analyzes the handler of the TCP layer and the HANDLER,TCP layer of the HTTP application layer handler is not available because the TCP layer is only responsible for transferring bytes, but does not know how to parse the bytes received, and how to handle the ; Therefore, the application-layer protocol, if the TCP protocol is to be used, must inherit the TCP handler and then implement the handle function; For example, the handler of the HTTP layer implements the handle function, parses the HTTP protocol, processes the business request, and returns the result to the client First look at the handler of the TCP layer

TCP Layer Handler

TCP layer handler mainly have Baserequesthandler and Streamrequesthandler (all in socketserver.py file), first look at the Baserequesthandler code,

Classbaserequesthandler:
def__init__ (self, request, client_address, server):
 self.request = Request
 self.client_address = client_address
 self.server = Server
 self.setup ()
try:
 self.handle ()
Finally:
 self.finish ()

defsetup (self):
pass

defhandle (self):
pass

deffinish (self) : Pass

Before looking at the server, knowing that processing the client request is done in the initialization function of the handler class, and by initializing the function by this base class, we know that the processing request probably goes through three processes:

1. Setup to the client socket to make some settings;
2. Handle the function that really handles the request;
3. Finish closes socket read and write request;
   

This baserequesthandler is the handler top level base class, which simply abstracts out the handler overall framework and does not actually deal with it; we look at the TCP handler,

Classstreamrequesthandler (baserequesthandler):
 timeout = None
 disable_nagle_algorithm = False

defsetup (self):
 self.connection = self.request
ifself.timeoutisnotnone:
 self.connection.settimeout ( Self.timeout)
ifself.disable_nagle_algorithm:
 self.connection.setsockopt (socket. IPPROTO_TCP,
 socket. Tcp_nodelay, True)
 self.rfile = Self.connection.makefile (' RB ', self.rbufsize)
 self.wfile = Self.connection.makefile (' WB ', self.wbufsize)

deffinish (self):
ifnotself.wfile.closed:
try:
 Self.wfile.flush ()
exceptsocket.error:
pass
 self.wfile.close ()
 self.rfile.close ()

The TCP handler implements the setup and finish functions, the Setup function sets the timeout time, opens the Nagle algorithm, and sets the socket read-write cache, and the finish function closes the socket read-write;

From the handler of the above two TCP layers, to implement a server handler based on HTTP, it is necessary to inherit the Streamrequesthandler class and realize handle function, so this is the main thing that HTTP layer handler do;

HTTP Layer Handler

From the previous TCP layer handler, we know that the HTTP layer handler on the basis of inheriting the TCP layer handler, mainly realizes the handle function processing client's request, or directly read the code;

Defhandle (self):
 self.close_connection = True

 self.handle_one_request ()
Whilenotself.close_ Connection:
 self.handle_one_request ()

This is the handle function of Basehttprequesthandler, where the handle function calls the Handle_one_request function to process a request, which is a short link by default, so after a request has been executed, Will not enter the while loop processes the next request on the same connection, but it is judged within the Handle_one_request function if the connection in the request header is Keep_ Alive or HTTP version is greater than or equal to 1.1, you can maintain long links, and then look at how the Handle_one_request function is handled;

Defhandle_one_request (self):
try:
self.raw_requestline =self.rfile.readline (65537)
Iflen (self.raw_ Requestline) >65536:
self.requestline = '
self.request_version = '
self.command
= ' Self.send_ Error (Httpstatus.request_uri_too_long)
return
ifnotself.raw_requestline:
self.close_connection = True
return
ifnotself.parse_request ():
return
 mname = ' do_ ' +self.command
ifnothasattr ( Self, mname):
self.send_error (
 httpstatus.not_implemented,
"Unsupported Method (%r)"%self.command) return method
 = GetAttr (self, Mname) method
 ()
Self.wfile.flush ()
 except Socket.timeout as E:
self.log_error ("Request timed out:%r", e)
self.close_connection = True

This handle_one_request execution process is as follows:

1. First call Parse_request parse client HTTP request content
2. Construct the function method for the request by "Do_" +command
3. Call the method function to process the business and return response to the client
   

This basehttprequesthandler is an HTTP handler base class and therefore cannot be used directly because it does not define a request-handling function, that is, the method function Fortunately Python provides us with a simple simplehttprequesthandler that inherits the Basehttprequesthandler and implements the request function; Let's look at the Get function:

# simplehttprequesthandler
#---------------------------------------------
defdo_get (self): "" "
Serve A GET request.
 "" " f = self.send_head ()
iff:
try:
 self.copyfile (F, Self.wfile)
finally:
 f.close ()

This get function first calls the Do_get function to return the response header to the client and returns the requested file, and finally calls the CopyFile function to return the request file through the connection to the client;

The above is the most basic content of HTTP module, finally, the summary of the example program handler part:

1. Server passes the request to the Simplehttprequesthandler initialization function;
2. Simplehttprequesthandler in the initialization section, the client connection to make some settings;
3. Then call the handle function to process the request;
4. The handle function then invokes the Handle_one_request processing request;
5. Within the Handle_one_request function, parse the request and find the request processing function;
6. My previous access belongs to get access, so call the Do_get function directly to return the index.html file to the client;
   

The Python HTTP module has been parsed to this end; I don't know if you've noticed that Python's own HTTP module is not very convenient to use because it calls the request function through the request method, so that when the same method is called a lot of times, such as Get and post methods, can cause this request function to be extremely large, the code is not good to write, each kind of situation judge; Of course, Simplehttprequesthandler is just a simple example provided by Python;

Of course, the Python authorities provide a better framework for HTTP, namely WSGI server and WSGI application, and the next article analyzes Python's own wsgiref modules and bottle, followed by the analysis of flask;