Use Python to compile a super guide for HTTP servers

This article describes how to use Python to compile a super guide for HTTP servers. It also describes how to compile a web server based on the Python framework, recommended! For more information, see what is a network server first?

In short, it is a network connection server built on a physical server, waiting for the client to send requests permanently. When the server receives the request, it generates a response and returns it to the client. Communication between the client and the server is based on the HTTP protocol. The client can be a browser or any software that supports HTTP.

What is the simple implementation form of the network server? Below is my understanding of this. The sample code is implemented in the Python language. However, even if you do not understand the Python language, you can understand the related concepts from the code and the following explanations:

import socketHOST, PORT = '', 8888listen_socket = socket.socket(socket.AF_INET, socket.SOCK_STREAM)listen_socket.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)listen_socket.bind((HOST, PORT))listen_socket.listen(1)print 'Serving HTTP on port %s ...' % PORTwhile True:  client_connection, client_address = listen_socket.accept()  request = client_connection.recv(1024)  print request  http_response = """\  HTTP/1.1 200 OK  Hello, World!  """  client_connection.sendall(http_response)  client_connection.close()

Save the above Code as webserver1.py, or download it directly from my [Github repository] (https://github.com/rspivak/lsbaws/ B lob/master/part1/webserver1.py), and then run the file through the command line:

$ python webserver1.pyServing HTTP on port 8888 …

Next, enter the link http: // localhost: 8888/hello in the address bar of the browser, and press enter to see the magic scene. "Hello, World!" should appear in the browser !" This sentence:

Is it amazing? Next, let's analyze the implementation principles behind it.

First, let's look at the network address you entered. Its name is URL (Uniform Resource Locator, unified Resource Locator). Its basic structure is as follows:

Through the URL, you tell the Web server address that the browser needs to discover and connect to, and get the page path on the server. However, before the browser sends an HTTP request, it must first establish a tc p connection with the target network server. Then, the browser sends an HTTP request to the server through a TCP connection and waits for the server to return an HTTP response. When the browser receives a response, the response content is displayed on the page.
The browser displays "Hello, World !" This sentence.

So how does one establish a TCP connection before the client sends a request and the server returns a response? To establish a TCP connection, both the server and client use the so-called socket ). Next, we do not directly use the browser, but manually simulate the browser using telnet on the command line.

On the same computer that runs the network server, enable a telnet session through the command line, set the host to be connected to as localhost, set the host's connection port to 8888, and then press the Enter key:

$ telnet localhost 8888Trying 127.0.0.1 …Connected to localhost.

After completing these operations, you have established a TCP connection with the local running network server to send and receive HTTP information at any time. The following figure shows the standard process required by the server to accept the new TCP connection.

In the telnet session above, enter GET/hello HTTP/1.1 and press Enter:

$ telnet localhost 8888Trying 127.0.0.1 …Connected to localhost.GET /hello HTTP/1.1HTTP/1.1 200 OKHello, World!

You have successfully simulated the browser manually! You manually send an HTTP request and receive an HTTP response. The following figure shows the basic structure of the HTTP request:

The HTTP request line includes the HTTP method (the GET method is used here because we want to obtain the content from the server), the server page path (/hello), and the HTTP protocol version.

To simplify it as much as possible, the current network server does not parse the above request. You can enter code that is meaningless or receive "Hello, World! "Response.

After you enter the Request Code and press the Enter key, the client sends the request to the server. The server will parse the request and return the corresponding HTTP response.

The following figure shows the HTTP Response Details returned from the server to the client:

Let's analyze it. The response contains the status line HTTP/1.1 200 OK, followed by a required blank line, followed by the body of the HTTP response.

The response status line HTTP/1.1 200 OK contains the HTTP Version, HTTP status code, and the Reason Phrase corresponding to the status code (Reason Phrase ). After the browser receives the response, the body of the response is displayed, Which is why "Hello, World!" is displayed in the browser !" This sentence.

This is how network servers work. A Brief Review: The network server first creates a listener socket (listening socket) and enables a perpetual cycle to receive new connections. After the client initiates a TCP connection with the server and successfully establishes a connection, after an HTTP request is sent to the server, the server returns an HTTP response. To establish a TCP connection, both the client and server use sockets.

Now that you have a basic available simple network server, you can use a browser or other HTTP client for testing. As shown above, you can use the telnet command and manually enter HTTP. You can also become an HTTP client.

Next, let's think about how to run the Djando application, Flask application, and Pyramid application on the server without making any changes to the server code, and meet the requirements of these different network frameworks?
Previously, the Python network framework you selected would restrict the network servers that can be used, and vice versa. If the framework and server can match each other during design, you will not face this problem:

However, if you try to combine servers and frameworks with unmatched designs, you will surely encounter the problem shown in the figure below:

This means that you can only use a combination of servers and frameworks that can run normally, rather than the server or framework you want to use.

So, how do you ensure that you can use your selected server without modifying the network server code or network framework code and match multiple different network frameworks? To solve this problem, the Python Web Server Gateway Interface ("WSGI") emerged ").

The emergence of WSGI allows developers to separate the network framework from the network server, without mutual restrictions. Now, you can mix different network servers and network development frameworks and choose a combination that meets your needs. For example, you can use Gunicorn, Nginx/uWSGI, or Waitress server to run Django, Flask, or Pyramid applications. It is precisely because the server and framework support WSGI that a free mix of the two can be achieved.

Therefore, WSGI is the answer to the questions I have left in the previous article. Your network server must implement a server-side WSGI interface. Currently, all modern Python network frameworks have implemented the WSGI interface of the framework, so developers do not need to modify the server code, A network framework is supported.

The network server and network framework support the WSGI protocol, which not only allows application developers to choose a combination that meets their needs, but also facilitates server and framework developers, because they can focus on what they are good at, rather than focusing on each other. Other programming languages also have similar interfaces: Java Servlet APIs and Ruby Rack.

I guess you are thinking, "no code, no truth !" In this case, I will provide a very simple WSGI server implementation:

# Tested with Python 2.7.9, Linux & Mac OS Ximport socketimport StringIOimport sysclass WSGIServer(object):  address_family = socket.AF_INET  socket_type = socket.SOCK_STREAM  request_queue_size = 1  def __init__(self, server_address):    # Create a listening socket    self.listen_socket = listen_socket = socket.socket(      self.address_family,      self.socket_type    )    # Allow to reuse the same address    listen_socket.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)    # Bind    listen_socket.bind(server_address)    # Activate    listen_socket.listen(self.request_queue_size)    # Get server host name and port    host, port = self.listen_socket.getsockname()[:2]    self.server_name = socket.getfqdn(host)    self.server_port = port    # Return headers set by Web framework/Web application    self.headers_set = []  def set_app(self, application):    self.application = application  def serve_forever(self):    listen_socket = self.listen_socket    while True:      # New client connection      self.client_connection, client_address = listen_socket.accept()      # Handle one request and close the client connection. Then      # loop over to wait for another client connection      self.handle_one_request()  def handle_one_request(self):    self.request_data = request_data = self.client_connection.recv(1024)    # Print formatted request data a la 'curl -v'    print(''.join(      '< {line}\n'.format(line=line)      for line in request_data.splitlines()    ))    self.parse_request(request_data)    # Construct environment dictionary using request data    env = self.get_environ()    # It's time to call our application callable and get    # back a result that will become HTTP response body    result = self.application(env, self.start_response)    # Construct a response and send it back to the client    self.finish_response(result)  def parse_request(self, text):    request_line = text.splitlines()[0]    request_line = request_line.rstrip('\r\n')    # Break down the request line into components    (self.request_method, # GET     self.path,      # /hello     self.request_version # HTTP/1.1     ) = request_line.split()  def get_environ(self):    env = {}    # The following code snippet does not follow PEP8 conventions    # but it's formatted the way it is for demonstration purposes    # to emphasize the required variables and their values    #    # Required WSGI variables    env['wsgi.version']   = (1, 0)    env['wsgi.url_scheme']  = 'http'    env['wsgi.input']    = StringIO.StringIO(self.request_data)    env['wsgi.errors']    = sys.stderr    env['wsgi.multithread'] = False    env['wsgi.multiprocess'] = False    env['wsgi.run_once']   = False    # Required CGI variables    env['REQUEST_METHOD']  = self.request_method  # GET    env['PATH_INFO']     = self.path       # /hello    env['SERVER_NAME']    = self.server_name    # localhost    env['SERVER_PORT']    = str(self.server_port) # 8888    return env  def start_response(self, status, response_headers, exc_info=None):    # Add necessary server headers    server_headers = [      ('Date', 'Tue, 31 Mar 2015 12:54:48 GMT'),      ('Server', 'WSGIServer 0.2'),    ]    self.headers_set = [status, response_headers + server_headers]    # To adhere to WSGI specification the start_response must return    # a 'write' callable. We simplicity's sake we'll ignore that detail    # for now.    # return self.finish_response  def finish_response(self, result):    try:      status, response_headers = self.headers_set      response = 'HTTP/1.1 {status}\r\n'.format(status=status)      for header in response_headers:        response += '{0}: {1}\r\n'.format(*header)      response += '\r\n'      for data in result:        response += data      # Print formatted response data a la 'curl -v'      print(''.join(        '> {line}\n'.format(line=line)        for line in response.splitlines()      ))      self.client_connection.sendall(response)    finally:      self.client_connection.close()SERVER_ADDRESS = (HOST, PORT) = '', 8888def make_server(server_address, application):  server = WSGIServer(server_address)  server.set_app(application)  return serverif __name__ == '__main__':  if len(sys.argv) < 2:    sys.exit('Provide a WSGI application object as module:callable')  app_path = sys.argv[1]  module, application = app_path.split(':')  module = __import__(module)  application = getattr(module, application)  httpd = make_server(SERVER_ADDRESS, application)  print('WSGIServer: Serving HTTP on port {port} ...\n'.format(port=PORT))  httpd.serve_forever()

The above code is much longer than the first part of the server implementation code, but the code is not too long. There are only less than 150 lines. It is not difficult to understand. The above server has more features-it can run network applications written by you using your favorite framework, whether you choose Pyramid, Flask, Django, or other frameworks that support WSGI protocol.

You don't believe it? You can test the result by yourself. Save the above Code as webserver2.py or download it directly from my Github repository. If you do not provide any parameters when running the file, the program reports an error and exits.

$ python webserver2.pyProvide a WSGI application object as module:callable

The purpose of the above program design is to run the network application you developed, but you still need to meet some of its requirements. To run the server, you only need to install Python. However, to run network applications developed using frameworks such as Pyramid, Flask, and Django, you need to install these frameworks first. Next we will install these three frameworks. I prefer to use virtualenv for installation. Follow the prompts below to create and activate a virtual environment and install the three network frameworks.

$ [sudo] pip install virtualenv$ mkdir ~/envs$ virtualenv ~/envs/lsbaws/$ cd ~/envs/lsbaws/$ lsbin include lib$ source bin/activate(lsbaws) $ pip install pyramid(lsbaws) $ pip install flask(lsbaws) $ pip install django

Next, you need to create a network application. First, create the Pyramid application. Save the following code as a pyramidapp. py file, put it in the folder where webserver2.py is located, or download the file directly from my Github Repository:

from pyramid.config import Configuratorfrom pyramid.response import Responsedef hello_world(request):  return Response(    'Hello world from Pyramid!\n',    content_type='text/plain',  )config = Configurator()config.add_route('hello', '/hello')config.add_view(hello_world, route_name='hello')app = config.make_wsgi_app()

Now, you can start the above Pyramid application through a self-developed network server.

(lsbaws) $ python webserver2.py pyramidapp:appWSGIServer: Serving HTTP on port 8888 ...

When running webserver2.py, you tell your server to load the callable object of the app in the pyramidapp module ). Your server can now receive HTTP requests and transfer the requests to your Pyramid application. Currently, an application can only process one route:/hello. Enter http: // localhost: 8888/hello in the address bar of the browser, and press enter to observe what will happen:

You can also use the curl command on the command line to test the server running status:

$ curl -v http://localhost:8888/hello...

Next, we will create the Flask application. Repeat the preceding steps.

from flask import Flaskfrom flask import Responseflask_app = Flask('flaskapp')@flask_app.route('/hello')def hello_world():  return Response(    'Hello world from Flask!\n',    mimetype='text/plain'  )app = flask_app.wsgi_app

Save the above Code as flaskapp. py, or directly download the file from my Github repository and run:

(lsbaws) $ python webserver2.py flaskapp:appWSGIServer: Serving HTTP on port 8888 ...

Enter http: // localhost: 8888/hello in the address bar of the browser, and press Enter:

Similarly, use the curl command on the command line to check whether the server will return the information generated by the Flask application:

$ curl -v http://localhost:8888/hello...

Does this server support Django applications? Just try it! However, the subsequent operations are more complex. We recommend that you clone the entire repository and use the djangoapp. py file. The following code adds a Django application named helloworld to the current Python path and imports the WSGI application of the project.

import syssys.path.insert(0, './helloworld')from helloworld import wsgiapp = wsgi.application

Save the above Code as djangoapp. py and run the Django application on the server you developed.

(lsbaws) $ python webserver2.py djangoapp:appWSGIServer: Serving HTTP on port 8888 ...

Similarly, enter http: // localhost: 8888/hello in the browser and press Enter:

Next, like the previous tests, you run the curl command through the command line to verify that the Djando application has successfully processed your request:

$ curl -v http://localhost:8888/hello...

Have you performed the test according to the above steps? Have you enabled the server to support all three frameworks? If not, try your own operations. It is important to read the code, but the purpose of this series of articles is re-development, which means you need to do it yourself. It is best to re-enter all the code and make sure that the code running result meets expectations.

After the above introduction, you should have known the power of WSGI: it allows you to freely mix network servers and frameworks. WSGI provides a simple interface for the interaction between the Python network server and the Python network framework, and is very easy to implement on the server and framework. The following code snippet shows the WSGI interfaces on the server and framework respectively:

def run_application(application):  """Server code."""  # This is where an application/framework stores  # an HTTP status and HTTP response headers for the server  # to transmit to the client  headers_set = []  # Environment dictionary with WSGI/CGI variables  environ = {}  def start_response(status, response_headers, exc_info=None):    headers_set[:] = [status, response_headers]  # Server invokes the ‘application' callable and gets back the  # response body  result = application(environ, start_response)  # Server builds an HTTP response and transmits it to the client  …def app(environ, start_response):  """A barebones WSGI app."""  start_response('200 OK', [('Content-Type', 'text/plain')])  return ['Hello world!']run_application(app)

The following explains how the above Code works:

The network framework provides a callable object named application (WSGI does not specify how to implement this object ).
The server calls the application every time it receives a request from the HTTP client. It will pass an environ dictionary as a parameter to the callable object, which contains many variables of WSGI/CGI and a callable object named start_response.
The framework/application generates the HTTP status code and HTTP response Header (HTTP response headers), and then transmits the two to start_response, waiting for the server to save. In addition, the framework/application will return the response body.
The server combines the status code, response header, and response body into an HTTP Response and returns it to the client (this step is not part of the WSGI protocol ).
The following figure intuitively shows the WSGI interface:

One thing to remind everyone is that when you use the above framework to develop network applications, you are dealing with a higher level of logic and will not directly handle WSGI protocol-related requirements, but I am very clear, since you are reading this article, you must be interested in the WSGI interface of the framework. Therefore, without using the Pyramid, Flask, or Djando framework, we will develop a simple WSGI network application/network framework and use the WSGI server to run the application:

def app(environ, start_response):  """A barebones WSGI application.  This is a starting point for your own Web framework :)  """  status = '200 OK'  response_headers = [('Content-Type', 'text/plain')]  start_response(status, response_headers)  return ['Hello world from a simple WSGI application!\n']

Save the above Code as a wsgiapp. py file, or download it directly from my Github repository, and then run the application on the network server:

(lsbaws) $ python webserver2.py wsgiapp:appWSGIServer: Serving HTTP on port 8888 ...

Enter the address in the browser and press Enter. The result is as follows:

You have just compiled a simple WSGI network framework! That's incredible.

Next, we re-analyze the objects that the server returns to the client. The following figure shows the HTTP response generated by the server after you call the Pyramid application through the HTTP client:

The response in is similar to what you saw in the first article, but there are also obvious differences. For example, four HTTP headers: Content-Type, Content-Length, Date, and Server that you have never seen before are displayed. The response objects returned by the network server usually contain headers. However, none of the four are required. The header is used to transmit additional information about HTTP requests/responses.

Since you have a deeper understanding of the WSGI interface, the following figure gives a more detailed explanation of the content of the response object, explaining how each content is generated.

So far, I have not introduced the specific content of the environ dictionary, but simply put, it must contain some WSGI and CGI variables specified by the WSGI protocol. The server obtains the value required by the dictionary from the HTTP request. The following figure shows the dictionary details:

The network framework uses the information provided by this dictionary to determine which view to use, where to read the Request body, and how to output error information based on specified routing and request methods.

Up to now, you have successfully created your own network servers that support the WSGI protocol, and developed multiple network applications using different network frameworks. In addition, you have developed a simple network framework. The content described in this article is not very rich. Next, let's review how the WSGI network server processes HTTP requests:

• First, the server starts and loads the application callable object provided by the network framework/application.
• Then, the server reads a request information
• The server then parses the request
• Then, the server uses the request data to create a dictionary named environ.
• Then, the server uses the environ dictionary and start_response to call the application as parameters and obtain the response body generated by the application.
• Then, the server constructs an HTTP response based on the data returned after the application object is called, the status code and Response Header set by start_response.
• Finally, the server returns the HTTP Response to the client.

The above is all the content in the second part. You now have a running WSGI server that supports network applications written by a network framework that complies with the WSGI protocol. Best of all, this server can work with multiple network frameworks without any code modifications.