Python socket module source code implementation Points Analysis, pythonsocket

Python socket module source code implementation Points Analysis, pythonsocket

BaseServer and BaseRequestHandler
Python improves more advanced encapsulation for network programming. SocketServer. py provides many network services. They are elegantly designed. Python abstracts network services into two main classes. One is the Server class, which is used to process connection-related network operations, and the other is the RequestHandler class, which is used to process data-related operations. Two MixIn classes are provided to expand the Server and implement multi-process or multi-thread. When building a network service, the Server and RequestHandler are not separated. The instance object of RequestHandler works with the Server in the Server.

The main Server relationships of the module are as follows:

    +------------+    | BaseServer |    +------------+       |       v    +-----------+    +------------------+    | TCPServer |------->| UnixStreamServer |    +-----------+    +------------------+       |       v    +-----------+    +--------------------+    | UDPServer |------->| UnixDatagramServer |    +-----------+    +--------------------+

BaseServer Analysis
BaseServer provides the serve_forever and handler_request Methods externally through _ init _ initialization.

Init initialization:

  def __init__(self, server_address, RequestHandlerClass):    """Constructor. May be extended, do not override."""    self.server_address = server_address    self.RequestHandlerClass = RequestHandlerClass    self.__is_shut_down = threading.Event()    self.__shutdown_request = False

_ Init _ source code is very simple. The main function is to create a server object, initialize the server address, and process the requested class. Familiar with socket programming, it should be clear that server_address is a tuple containing host and port.

Serve_forever
After the server object is created, you need to use the server object to open an infinite loop. The following analyzes the source code of serve_forever.

  def serve_forever(self, poll_interval=0.5):    self.__is_shut_down.clear()    try:      while not self.__shutdown_request:        r, w, e = _eintr_retry(select.select, [self], [], [],                    poll_interval)        if self in r:          self._handle_request_noblock()    finally:      self.__shutdown_request = False      self.__is_shut_down.set()

Serve_forever accepts the poll_interval parameter to indicate the select round-robin time. Then it enters an infinite loop and calls the select method for network I/O listening.

If the select function returns an IO connection or data, the _ handle_request_noblock method is called.

_handle_request_noblock  def _handle_request_noblock(self):    try:      request, client_address = self.get_request()    except socket.error:      return    if self.verify_request(request, client_address):      try:        self.process_request(request, client_address)      except:        self.handle_error(request, client_address)        self.shutdown_request(request)

The _ handle_request_noblock method starts to process a request and is not blocked. This method gets the connection through the get_request method. The specific implementation is in its subclass. Once a connection is obtained, call the verify_request method to verify the request. If the verification succeeds, process_request is called to process the request. If an error occurs halfway, handle_error is called to handle the error and shutdown_request is used to end the connection.

verify_request  def verify_request(self, request, client_address):    return True

This method is used to verify the request, which is usually overwritten by the quilt class. Simply return True, and then go to the process_request method to process the request.

process_request  def process_request(self, request, client_address):    self.finish_request(request, client_address)    self.shutdown_request(request)

The process_request method is the entry of mixin. The MixIn subclass overrides this method to configure multithreading or multi-process. Call finish_request to process the request and call shutdown_request to end the request.

finish_request  def finish_request(self, request, client_address):    self.RequestHandlerClass(request, client_address, self)

The finish_request method will process the request. Create a requestHandler object and perform specific processing through requestHandler.

BaseRequestHandler Analysis
All requestHandler inherit the BaseRequestHandler base class.

  def __init__(self, request, client_address, server):    self.request = request    self.client_address = client_address    self.server = server    self.setup()    try:      self.handle()    finally:      self.finish()

This class processes each request. Set the request object when initializing the object. Then call the setup method. The subclass will rewrite this method to process socket connections. The following describes the handler and finish methods. You can override the handler method to process all requests.

Now, the Server method provided by Python is described. To sum up, to build a network service, a BaseServer is required to process network IO, and a requestHandler object is created internally to process all specific requests.

BaseServer-BaseRequestHandler

__init__(server_address, RequestHandlerClass):   BaseServer.server_address  BaseServer.RequestHandlerClassserve_forever():   select()   BaseServer._handle_request_noblock()    BaseServer.get_request() -> request, client_addres    BaseServer.verify_request()      BaseServer.process_request()        BaseServer.process_request()          BaseServer.finish_request()            BaseServer.RequestHandlerClass()              BaseRequestHandler.__init__(request)                BaseRequestHandler.request                BaseRequestHandler.client_address = client_address                BaseRequestHandler.setup()                BaseRequestHandler.handle()          BaseServer.shutdown_request()            BaseServer.close_request()      BaseServer.shutdown_request()        BaseServer.close_request()

BaseServer and BaseRequestHandler are two base classes for network processing. In actual applications, most network operations use TCP or HTTP. SocketServer. py also provides more advanced TCP and UDP encapsulation. Next let's take a look at the TCP network module (the difference between UDP and TCP in the Code organization is not very big, and it is ignored for the moment ).

TCPServer
The TCPServer inherits the BaseServer. during initialization, the socket is created.

def __init__(self, server_address, RequestHandlerClass, bind_and_activate=True):  BaseServer.__init__(self, server_address, RequestHandlerClass)  self.socket = socket.socket(self.address_family,                self.socket_type)  if bind_and_activate:    self.server_bind()    self.server_activate()

The _ init _ method creates a socket object through the socket module and then calls server_bind and server_activate.

server_binddef server_bind(self):  if self.allow_reuse_address:    self.socket.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)  self.socket.bind(self.server_address)  self.server_address = self.socket.getsockname()

The server_bind method performs the bind operation on the socket object and sets socket-related attributes, such as network address multiplexing.

server_activatedef server_activate(self):  self.socket.listen(self.request_queue_size)

Server_activate method is also relatively simple, adding the listen of the socket object.

Get_request
The most important method of this class is get_request. This method returns the socket Object Request connection.

  def get_request(self):  """Get the request and client address from the socket.  """  return self.socket.accept()

The get_request method is called in _ handle_request_noblock of the BaseServer base class, where the connection information obtained by the socket object is passed in. For UDPServer, UDP connections are obtained here.

In addition, TCPServer also provides a fileno method to return the file descriptor for the select call of the base class.

StreamRequestHandler
TCPServer implements the network service using tcp sockets, while the Handler is the corresponding StreamRequestHandler. It inherits BaseRequestHandler. The setup method and finish method of the base class are overwritten by this method to implement read/write operations on cached files through connections.

Setup method:

def setup(self):  self.connection = self.request  if self.timeout is not None:    self.connection.settimeout(self.timeout)  if self.disable_nagle_algorithm:    self.connection.setsockopt(socket.IPPROTO_TCP,                  socket.TCP_NODELAY, True)  self.rfile = self.connection.makefile('rb', self.rbufsize)  self.wfile = self.connection.makefile('wb', self.wbufsize)

Setup determines whether the nagle algorithm is used. Then set the corresponding connection property. The most important thing is to create a readable (rfile) and a writable "file" object. They didn't actually create a file, instead, it encapsulates the operations for reading and sending data and abstracts them into operations on files. It can be understood that self. rfile is the object for reading client data. It has some methods to read data. Self. wfile is the object used to send data to the client. In subsequent operations, the client data will be read by the write buffer. When sending data to the client, you only need to write data to writable files.

To implement TCP services, you must use TCPServer and StreamRequestHandler to work together. The procedure for calling a function is as follows. A function call is represented by parentheses. A value assignment is not enclosed by parentheses. A value without a class prefix indicates a system call:

TCPServer-StreamRequestHandler

__init__(server_address, RequestHandlerClass):   BaseServer.server_address  BaseServer.RequestHandlerClass  TCPServer.socket = socket.socket(self.address_family, self.socket_type)  TCPServer.server_bind()  TCPServer.server_activate()serve_forever():   select()   BaseServer._handle_request_noblock()    TCPServer.get_request() -> request, client_addres      socket.accept()    BaseServer.verify_request()      BaseServer.process_request()        BaseServer.process_request()          BaseServer.finish_request(request, client_address)            BaseServer.RequestHandlerClass()              BaseRequestHandler.__init__(request)                BaseRequestHandler.request                BaseRequestHandler.client_address = client_address                StreamRequestHandler.setup()                  StreamRequestHandler.connection = StreamRequestHandler.request                  StreamRequestHandler.rfile                  StreamRequestHandler.wfile                BaseRequestHandler.handle()                StreamRequestHandler.finsih()                  StreamRequestHandler.wfile.close()                  StreamRequestHandler.rfile.close()          BaseServer.shutdown_request(request)            TCPServer.shutdown()              request.shutdown()            TCPServer.close_request(request)              request.close()      TCPServer.shutdown_request(request)        TCPServer.shutdown(request)          request.shutdown()        TCPServer.close_request(request)          request.close()

When we first introduced BaseServer, we knew that python reserved interfaces that can be used for Mixin extension of multithreading or multi-process during BaseServer design. Mixin is implemented by the parse_request method of the parent class.

ThreadingMixIn
The ThreadingMixIn class implements multiple threads. It has only two methods: process_request and process_request_thread. The multi-process method is ForkingMixIn, Which is skipped for the moment.

process_requestdef process_request(self, request, client_address):  t = threading.Thread(target = self.process_request_thread,             args = (request, client_address))  t.daemon = self.daemon_threads  t.start()

The process_request method returns the method of the parent class. This serves as the interface entry to open a new Thread for each request by calling the Thread. Each thread is bound to the process_request_thread method.

process_request_thread  def process_request_thread(self, request, client_address):    try:      self.finish_request(request, client_address)      self.shutdown_request(request)    except:      self.handle_error(request, client_address)      self.shutdown_request(request)

The process_request_thread method is almost the same as the parse_request method in BaseServer. It is called in multiple threads.

When used, the interface is called through multiple inheritance methods, for example:

class ThreadingTCPServer(ThreadingMixIn, TCPServer):   pass

The specific call process is as follows:

ThreadingMixIn -- TCPServer - StreamRequestHandler__init__(server_address, RequestHandlerClass):   BaseServer.server_address  BaseServer.RequestHandlerClass  TCPServer.socket = socket.socket(self.address_family, self.socket_type)  TCPServer.server_bind()  TCPServer.server_activate()serve_forever():   select()   BaseServer._handle_request_noblock()    TCPServer.get_request() -> request, client_addres      socket.accept()    BaseServer.verify_request()      BaseServer.process_request()        ThreadingMixIn.process_request()          t = threading.Thread(target = ThreadingMixIn.process_request_thread)          ThreadingMixIn.process_request_thread            BaseServer.finish_request(request, client_address)              BaseServer.RequestHandlerClass()                BaseRequestHandler.__init__(request)                  BaseRequestHandler.request                  BaseRequestHandler.client_address = client_address                  StreamRequestHandler.setup()                    StreamRequestHandler.connection = StreamRequestHandler.request                    StreamRequestHandler.rfile                    StreamRequestHandler.wfile                  BaseRequestHandler.handle()                  StreamRequestHandler.finsih()                    StreamRequestHandler.wfile.close()                    StreamRequestHandler.rfile.close()            BaseServer.shutdown_request(request)              TCPServer.shutdown()                request.shutdown()              TCPServer.close_request(request)                request.close()      TCPServer.shutdown_request(request)        TCPServer.shutdown(request)          request.shutdown()        TCPServer.close_request(request)          request.close()