A super Guide to writing HTTP servers using Python

First, what exactly is a Web server?

In short, it is a network connection server (Networking server) that is built on a physical server, permanently waiting for the client to send the request. When the server receives the request, it generates a response and returns it to the client. The communication between the client and the server is done in the HTTP protocol. The client can be a browser or any software that supports the HTTP protocol.

So what is the simple implementation of a Web server? Here is my understanding of this. The sample code is implemented in the Python language, but even if you do not understand the Python language, you should also be able to understand the relevant concepts from the code and the following explanations:

Import sockethost, PORT = ', 8888listen_socket = Socket.socket (socket.af_inet, socket. SOCK_STREAM) listen_socket.setsockopt (socket. Sol_socket, SOCKET. SO_REUSEADDR, 1) listen_socket.bind ((HOST, PORT)) Listen_socket.listen (1) print ' serving HTTP on PORT%s ... '% portwhile Tr UE:  client_connection, client_address = listen_socket.accept ()  request = CLIENT_CONNECTION.RECV (1024x768)  Print request  http_response = "" \  http/1.1 OK  Hello, world!  "" " Client_connection.sendall (http_response)  client_connection.close ()

Save the above code as webserver1.py, or download it directly from my [GitHub warehouse] (https://github.com/rspivak/lsbaws/b lob/master/part1/webserver1.py), Then run the file from the command line:

$ python webserver1.pyserving HTTP on port 8888 ...

Next, enter the link Http://localhost:8888/hello in the address bar of the browser and press ENTER, and you will see a magical scene. In the browser, "Hello, world!" should appear This sentence:

Isn't it amazing? Next, let's analyze the underlying implementation principle.

First, let's look at the network address you entered. Its name is URL (Uniform Resource Locator, Uniform Resource Locator), and its basic structure is as follows:

Through the URL, you tell the browser what it needs to discover and connect to the network server address, and get the page path on the server. However, before the browser sends an HTTP request, it first establishes a TC P connection with the target network server. The browser then sends an HTTP request to the server over a TCP connection and waits for the server to return an HTTP response. When the browser receives a response, the content of the response is displayed on the page, and in the example above
Child, the browser shows "Hello, world!" This sentence.

So, before the client sends the request, and the server returns the response, how did the two make the TCP connection? To establish a TCP connection, both the server and the client use the so-called socket (socket). Next, instead of using the browser directly, we use Telnet to manually emulate the browser at the command line.

On the same computer that is running the network server, open a Telnet session from the command line, set the host to localhost with the connection port set to 8888, and then press ENTER:

$ telnet localhost 8888Trying 127.0.0.1 ... Connected to localhost.

After you've done this, you've actually established a TCP connection with the local running network server, ready to send and receive HTTP messages. In the following image, the standard process required for the server to accept a new TCP connection is shown.

In the above Telnet session, we enter Get/hello http/1.1 and press ENTER:

$ telnet localhost 8888Trying 127.0.0.1 ... Connected to localhost. Get/hello http/1.1http/1.1 Okhello, world!

You successfully simulated the browser manually! You manually sent an HTTP request and then received an HTTP response. The following diagram shows the basic structure of the HTTP request:

The HTTP request line includes the HTTP method (The Get method is used here, because we want to get the content from the server), the Server page path (/hello), and the version of the HTTP protocol.

To make it as simple as possible, the Web server we currently implement does not parse the above request, you can enter some code that does not make any sense, and you can receive "Hello, world!" Response.

After you enter the request code and press ENTER, the client sends the request to the server, the server resolves the request that you sent and returns the corresponding HTTP response.

The following figure shows the HTTP response details returned to the client by the server:

Let's analyze it for a moment. The response contains the status line http/1.1, followed by a blank line that is required, followed by the body of the HTTP response.

The status line of the response, http/1.1, contains the HTTP version, the HTTP status code, and the reason phrase corresponding to the status code (Reason Phrase). After the browser receives the response, the body of the response is displayed, which is why you will see "Hello, world!" in the browser This sentence.

This is how the network server basically works. A brief recap: The network server first creates a listening socket (listening socket) and opens a perpetual loop to receive the new connection; The client initiates a TCP connection to the server, sends an HTTP request to the server after the connection is successfully established, and the server returns an HTTP response. To establish a TCP connection, both the client and the server use sockets.

Now that you have a simple Web server that is basically available, you can test it using a browser or other HTTP client. As shown above, you can also become an HTTP client by using the Telnet command and entering the HTTP request manually.

Let's think about it: how can you run Djando, flask, and pyramid applications through this server without any modifications to the server code, while meeting the requirements of these different network frameworks?
Previously, the Python network framework you chose would limit the network servers you could use and vice versa. If the framework and the server are designed to match each other, you will not face this problem:

But if you're trying to combine a server with a framework that doesn't match the design, then you're bound to run into the problem shown in the following diagram:

This means that you can basically only use a server and framework that works properly, not the server or framework you want to use.

So how do you make sure you can use your chosen server without modifying your network server code or network framework code, and match multiple different network frameworks? To solve this problem, a Python Web server gateway Interface (Python Web servers gateways Interface, referred to as "WSGI") appears.

The advent of WSGI allows developers to separate the network framework from the choice of Web servers, no longer restricting each other. Now you can really mix and match different Web servers with network development frameworks and choose the combination that meets your needs. For example, you can use a gunicorn or Nginx/uwsgi or waitress server to run a Django, flask, or pyramid application. Because both the server and the framework support WSGI, it is true that the free mix and match between the two are realized.

So, Wsgi is the answer to the question I left in my last article. Your Web server must implement a server-side WSGI interface, and now all modern Python network frameworks have implemented the WSGI interface on the framework side, so that developers do not need to modify the server's code to support a network framework.

The Web server and network framework support the WSGI protocol, which not only allows application developers to choose a combination of their own needs, but also facilitates the developers of servers and frameworks, because they can focus on their areas of expertise rather than competing with each other. Other programming languages also have similar interfaces: Java's servlet API and Ruby's rack, for example.

Accreditations, I guess you're thinking: "No code, no truth!" "In that case, I'm here to give a very simple WSGI server implementation:

# Tested with Python 2.7.9, Linux & Mac OS ximport socketimport stringioimport sysclass wsgiserver (object): Address_f amily = socket.af_inet Socket_type = socket. Sock_stream request_queue_size = 1 def __init__ (self, server_address): # Create a listening socket SELF.LISTEN_SOC Ket = Listen_socket = Socket.socket (self.address_family, Self.socket_type) # Allow to reuse the same add Ress listen_socket.setsockopt (socket. Sol_socket, SOCKET. SO_REUSEADDR, 1) # Bind Listen_socket.bind (server_address) # Activate Listen_socket.listen (self.request_queue_ Size) # Get server host name and port host, port = Self.listen_socket.getsockname () [: 2] self.server_name = socket . Getfqdn (Host) Self.server_port = port # Return headers set by Web framework/web application Self.headers_set = [ ] def set_app (self, application): self.application = Application def serve_forever (self): Listen_socket = Self.lis Ten_socket while True: # New ClienT connection self.client_connection, client_address = Listen_socket.accept () # Handle one request and close the Client connection.    Then # Loop-to-wait for another client connection self.handle_one_request () def handle_one_request (self):    Self.request_data = Request_data = Self.client_connection.recv (1024x768) # Print formatted request data a la ' curl-v ' Print ('. Join ' (' < {line}\n '. Format (line=line) for line in Request_data.splitlines ())) Self.parse_req Uest (request_data) # Construct Environment dictionary using request data env = Self.get_environ () # It's Time to Call we application callable and get # Back a result this would become HTTP response body result = Self.application ( env, Self.start_response) # Construct a response and send it back to the client Self.finish_response (result) def PA Rse_request (self, text): Request_line = Text.splitlines () [0] request_line = Request_line.rstrip (' \ r \ n ') # break D Own THe request line to components (Self.request_method, # GET Self.path, #/hello Self.request_version # HTT p/1.1) = Request_line.split () def get_environ (self): env = {} # The following code snippet does not follow PEP 8 Conventions # But it's formatted the IT's it's for demonstration purposes # to emphasize the required variables an    D Their values # # Required WSGI variables env[' wsgi.version ' = (1, 0) env[' wsgi.url_scheme '] = ' http ' env[' wsgi.input ' = Stringio.stringio (self.request_data) env[' wsgi.errors '] = Sys.stderr env[' Wsgi.multithread '] = False env[' wsgi.multiprocess '] = False env[' wsgi.run_once '] = false # Required CGI variables env[' Reque St_method '] = self.request_method # GET env[' path_info '] = self.path #/hello env[' server_name '] = SE Lf.server_name # localhost env[' server_port '] = str (self.server_port) # 8888 return env def start_response (SE LF, status, Response_heaDERs, Exc_info=none): # Add Necessary Server Headers server_headers = [(' Date ', ' Tue, Mar 12:54:48 GMT '), (' Server ', ' wsgiserver 0.2 '),] self.headers_set = [status, Response_headers + server_headers] # to Adhe Re to WSGI specification the Start_response must return # a ' write ' callable.    We simplicity ' s sake we ' ll ignore that detail # for now.       # return Self.finish_response def finish_response (self, result): Try:status, response_headers = Self.headers_set Response = ' http/1.1 {status}\r\n '. Format (status=status) for headers in Response_headers:response + = ' {0 }: {1}\r\n '. Format (*header) response + = ' \ r \ n ' for data in Result:response + = data # Print Formatte D response Data a la ' curl-v ' Print ('. > {line}\n '. Format (line=line) for line in RESPONSE.SP Litlines ()) Self.client_connection.sendall (response) Finally:self.client_connection.close () Server_add RESS = (HOST, PORT) = ' ', 8888def make_server (server_address, application): Server = Wsgiserver (server_address) server.set _app (application) return serverif __name__ = = ' __main__ ': If Len (SYS.ARGV) < 2:sys.exit (' provide a WSGI applicati On object as Module:callable ') App_path = sys.argv[1] module, application = App_path.split (': ') module = __import__ (mod ule) application = GetAttr (module, application) httpd = Make_server (server_address, application) print (' Wsgiserver:ser Ving Tsun HTTP on port {port} ... \ n '. Format (port=port)) Httpd.serve_forever ()

The above code is much longer than the first part of the server implementation code, but the code is actually not too long, only less than 150 lines, it is not too difficult to understand. The server above has more features-it can run Web applications that you write using your favorite frameworks, whether you choose Pyramid, Flask, Django, or other frameworks that support WSGI protocols.

You don't believe me? You can test it yourself to see what the results are. Save the above code as webserver2.py, or download it directly from my GitHub repository. If you do not provide any parameters when you run the file, the program will error and exit.

$ python webserver2.pyprovide a WSGI Application object as Module:callable

The purpose of the above program design is to run the Web application you developed, but you also need to meet some of its requirements. To run the server, you only need to install Python. But to run Web applications developed using frameworks such as pyramid, Flask, and Django, you'll need to install these frameworks first. We'll install these three kinds of frameworks next. I prefer to use virtualenv installation. Follow the prompts below to create and activate a virtual environment, and then install the three network frameworks.

$ [sudo] pip install virtualenv$ mkdir ~/envs$ virtualenv ~/envs/lsbaws/$ cd ~/envs/lsbaws/$ lsbin include lib$ source bin /activate (lsbaws) $ pip install Pyramid (lsbaws) $ pip install flask (lsbaws) $ pip Install Django

Next, you need to create a web app. We first create the Pyramid app. Save the following code as a pyramidapp.py file, put it in the folder where webserver2.py resides, or download the file directly from my GitHub repository:

From pyramid.config import configuratorfrom pyramid.response import responsedef hello_world (Request):  return Response (    ' Hello World from pyramid!\n ',    content_type= ' Text/plain ',  ) config = Configurator () config.add_ Route (' Hello ', '/hello ') Config.add_view (hello_world, route_name= ' hello ') app = Config.make_wsgi_app ()

Now you can launch the Pyramid app from the Web server you've developed.

(lsbaws) $ python webserver2.py pyramidapp:appWSGIServer:Serving HTTP on port 8888 ...

When you run webserver2.py, you tell your server to load the app callable object (callable) in the Pyramidapp module. Your server can now receive HTTP requests and transfer requests to your pyramid app. The app can currently handle only one route:/hello. Enter Http://localhost:8888/hello in the address bar of the browser and press ENTER to see what happens:

You can also use the Curl command at the command line to test the server's performance:

$ curl-v Http://localhost:8888/hello ...

Next we create the Flask app. Repeat the above steps.

From flask import flaskfrom flask Import Responseflask_app = Flask (' Flaskapp ') @flask_app. Route ('/hello ') def Hello_world ():  return Response (    ' Hello World from flask!\n ',    mimetype= ' Text/plain '  ) app = Flask_app.wsgi_app

Save the above code as flaskapp.py, or download the file directly from my GitHub repository and run:

(lsbaws) $ python webserver2.py flaskapp:appWSGIServer:Serving HTTP on port 8888 ...

Then enter Http://localhost:8888/hello in the browser address bar and press ENTER:

Similarly, use the Curl command at the command line to see if the server will return information generated by the Flask app:

$ curl-v Http://localhost:8888/hello ...

Does this server also support Django applications? Try it and know it! But the next step is more complicated, and I recommend that you clone the entire repository and use the djangoapp.py file. The following code adds a Django app named HelloWorld to the current Python path, and then imports the Wsgi app for that project.

Import syssys.path.insert (0, './helloworld ') from HelloWorld import Wsgiapp = wsgi.application

Save the above code as djangoapp.py and run the Django app using the server you developed.

(lsbaws) $ python webserver2.py djangoapp:appWSGIServer:Serving HTTP on port 8888 ...

Similarly, enter Http://localhost:8888/hello in the browser and press ENTER:

Next, as in previous times, you test with the Curl command on the command line confirming that the Djando application successfully processed your request:

$ curl-v Http://localhost:8888/hello ...

Did you follow the above steps to test? Did you get the server to support all three frameworks? If not, try to do it yourself. Reading the code is important, but the purpose of this series of articles is to re-develop, which means you need to do it yourself. It's best to re-enter all your code yourself and make sure the code runs as expected.

With this introduction, you should have realized the power of WSGI: It allows you to mix and match Web servers and frameworks freely. WSGI provides a minimalist interface for the interaction between the Python Web server and the Python network framework, and is very easy to implement on both the server side and the framework side. The following code snippet shows the Wsgi interface on both the server side and the frame end, respectively:

def run_application (application): "" "  Server Code.  " "" # This is where a application/framework stores  # an HTTP status and HTTP response headers for the server  # to Tra Nsmit to the client  Headers_set = []  # Environment dictionary with wsgi/cgi variables  environ = {}  def STA Rt_response (Status, Response_headers, Exc_info=none):    headers_set[:] = [status, Response_headers]  # Server Invokes the ' application ' callable and gets back  the # response body  result = Application (environ, start_respons e)  # Server builds an HTTP response and transmits it to the client  ... def app (environ, start_response): "" "  A Barebones WSGI app.  "" " Start_response (' K OK ', [(' Content-type ', ' Text/plain ')])  return [' Hello world! '] Run_application (APP)

Let us explain how the above code works:

The network framework provides a callable object named application (the WSGI protocol does not specify how to implement this object).
Each time the server receives a request from an HTTP client, application is called. It passes a dictionary called Environ to the callable object as a parameter that contains many variables of the wsgi/cgi and a callable object named Start_response.
The framework/app generates HTTP status codes and HTTP response Headers (HTTP response headers), and then passes them to Start_response, waiting for the server to be saved. In addition, the framework/app returns the body of the response.
The server combines the status code, response header, and response body into an HTTP response and returns it to the client (this step is not part of the WSGI protocol).
The following diagram visually illustrates the situation of the Wsgi interface:

It is important to remind you that when you use the above framework to develop Web applications, you are dealing with higher-level logic and do not deal directly with the requirements of the WSGI protocol, but I am well aware that since you are reading this article, you must be interested in the Wsgi interface of the frame end. So, let's develop a minimalist WSGI network application/network framework without using the pyramid, flask, or djando framework, and run the app using a WSGI server:

def app (environ, start_response): "" "  A barebones WSGI application.  This was a starting point for your own Web framework:) ""  status = ' OK '  response_headers = [(' Content-typ E ', ' Text/plain ')]  start_response (status, Response_headers)  return [' Hello World from a simple WSGI application !\n ']

Save the above code as a wsgiapp.py file, or download it directly from my GitHub repository and run the app with a Web server:

(lsbaws) $ python webserver2.py wsgiapp:appWSGIServer:Serving HTTP on port 8888 ...

Enter the address in the browser and press ENTER. The result should be this:

You have just written yourself a minimalist WSGI network framework! It's unbelievable.

Next, we re-parse the object returned to the client by the server. The following diagram shows the HTTP response generated by the server after you invoke the Pyramid app through an HTTP client:

is somewhat similar to what you saw in the first article, but there are also obvious differences. For example, there are 4 songs you haven't seen before. HTTP header: Content-type,content-length,date and server. These things. The response object returned by the Web server will typically contain a header. However, none of these four is necessary. The purpose of the header is to pass additional information about the HTTP request/response.

Now that you have a deeper understanding of the Wsgi interface, the following diagram provides a more detailed explanation of the content of the response object, explaining how each piece of content is produced.

So far, I have not covered the specifics of the Environ dictionary, but simply, it is a must contain some WSGI and CGI variables specified by the WSGI protocol. The server gets the value required for the dictionary from the HTTP request. The following diagram shows the details of the dictionary:

The network framework uses the information provided by the dictionary to determine which view (views) to use, and where to read the request body, and how to output the error message, based on parameters such as the Routing and request method specified.

As of now, you have successfully created your own Web server to support the WSGI protocol, and have developed multiple network applications using different network frameworks. In addition, you have developed a minimalist network framework. The contents of this article are not very rich. Let's review how the WSGI network server handles HTTP requests:

• First, the server starts and loads the application callable object provided by the network framework/app
• The server then reads a request message
• The server then parses the request
• The server then uses the request data to create a dictionary called Environ.
• The server then invokes application with the Environ dictionary and Start_response callable object as arguments, and obtains the response body that the app generates.
• The server then constructs an HTTP response based on the data returned after the Application object is called, as well as the status code and response headers of the Start_response settings.
• Finally, the server returns the HTTP response to the client.

This is all part of the second section. You now have a functioning WSGI server that supports network applications written by a network framework that adheres to the WSGI protocol. Best of all, this server can be used in conjunction with multiple network frameworks without any code modifications.

