Python full stack development 10, network programming

Network programming is how to realize the communication between two computers in the program, and the communication between the two computers is actually two inter-process communication, the communication between processes is mainly through the socket (socket) to describe the IP address (host) and port (process) to achieve, so we learn network programming, Must learn socket

First, socket

The socket module is "open", "Read and write" "Off" for both the server side and the client socket, and a complete socket model diagram is shown

There are some different functions that read and write operation calls in Python, so let's take a look at the implementation of a full socket (TCP) on the server and the client

# server.pyimport socketserver_socket = Socket.socket () server_socket.bind (( ' 127.0.0.1 ', 8000,)) Server_socket.listen (4) while true:conn, address = Server_socket.accept () # Block Conn        . Sendall (Bytes (' Greetings from the server, hello ', encoding= ' Utf-8 ') # Py3 to first convert to bytes, PY2 does not need to convert while True:ret = CONN.RECV (1024) If STR (ret, encoding= ' utf-8 ') = = ' Q ': Break Conn.sendall (bytes (str (ret, encoding= ' utf-8 ') + ' good ', enc oding= ' Utf-8 ') # received the client sent something plus a good return to the client # Client.pyimport Socketclient_socket = Socket.socket () client_socket.connect ((    ' 127.0.0.1 ', 8000,)) ret = CLIENT_SOCKET.RECV (1024) # Blocked in print (str (ret, encoding= ' Utf-8 ')) while True: INP = input (' Please enter what you want to send: ') Client_socket.sendall (bytes (INP, encoding= ' utf-8 ')) if inp = = ' Q ': Break ret = CLIENT_SOCKET.RECV (1024x768) print (str (ret, encoding= ' utf-8 ')) Client_socket.close () 

We can see that the whole process is implemented exactly according to the above diagram, so we need to keep the process of the above diagram in mind when we are learning.

Second, Socket class

SK = Socket.socket (socket.af_inet,socket. sock_stream,0) parameter one: Address cluster socket.af_inet IPv4 (default) Socket.af_inet6 IPv6 Socket.af_unix can only be used for single UNIX system interprocess communication parameter two: type socket. Sock_stream streaming socket, for TCP (default) socket. SOCK_DGRAM datagram socket, for UDP parameter three: protocol 0 (default) protocol related to a specific address family, if 0, the system will automatically select an appropriate protocol Sk.bind (address) based on the format and socket category S.bind ( Address) binds sockets to addresses. The format of address addresses depends on the address family. Under Af_inet, address is represented as a tuple (host,port). Sk.listen starts listening for incoming connections.      The backlog specifies the maximum number of connections that can be suspended before a connection is rejected. The backlog equals 5, indicating that the kernel has received a connection request, but the server has not yet called accept to handle the maximum number of connections 5 This value cannot be infinite because the connection queue sk.setblocking (bool) is blocked in the kernel (default true). If set to False, then the accept and recv when no data, then the error. Sk.accept () Accepts the connection and returns (Conn,address), where Conn is a new socket object that can be used to receive and send data.　　Address is the location of the connection client. The socket that receives the TCP client's connection (blocked) wait for the connection to Sk.connect (address) to connect to the address. Generally, address is in the form of a tuple (Hostname,port) and returns a socket.error error if there is an error in the connection. SK.CONNECT_EX (address), except that there will be a return value, the connection succeeds when return 0, the connection fails when the return encoding, for example: 10061sk.close () Close socket SK.RECV (Bufsize[,flag]) accept the socket data. The data is returned as a string, and bufsize specifies the maximum quantity that can be received. Flag provides additional information about the message, which can usually be ignored. Sk.recvfrom (BUFSIZE[.FLAG]) is similar to recv (), but the return value is (data,address). Where data is the string that contains the received information, address is the socket addressing that sent the data. Sk.send (String[,flag]) sends data from a string to a connected socket. The return value is the number of bytes to send, which may be less than the byte size of the string. That is, the specified content may not be sent all. Sk.sendall (String[,flag]) sends the data in a string to the connected socket, but attempts to send all the data before returning.      Successful return none, Failure throws an exception. Internally, the send is called recursively, sending all the content. Sk.sendto (string[,flag],address) sends data to the socket, address is a tuple in the form of (Ipaddr,port), specifying the remote address. The return value is the number of bytes sent. This function is mainly used for UDP protocol. Sk.settimeout (timeout) sets the timeout period for the socket operation, and timeout is a floating-point number in seconds. A value of None indicates no over-time. In general, hyper-times should be set when a socket is just created, because they may be used for connected operations (such as client connections waiting up to 5s) Sk.getpeername () to return the remote address of the connection socket. The return value is typically a tuple (ipaddr,port). Sk.getsockname () returns the socket's own address. Typically a tuple (ipaddr,port) Sk.fileno () a file descriptor for a socket

Third, set the socket

SetSockOpt () and getsockopt (), one is the SET option, and the other is the one that gets set. Here the main use is setsockopt (), setsockopt (Level,optname,value), level defines which option will be used. Usually it is sol_socket, which means the SOCKET option is being used.

The optname parameter provides special options for use. The settings for the available options vary slightly depending on the operating system. If level Sol_socket is selected, some common options are shown in the following table:

Options	Significance	Expectations
So_bindtodevice	You can make the socket valid only on a particular network interface (NIC). Maybe not a mobile portable device.	A string gives the name of the device or an empty string to return the default value
So_broadcast	Allow broadcast addresses to send and receive packets. Valid only for UDP. How to send and receive broadcast packets	Boolean integer
So_dontroute	Sending packets out through routers and gateways is prohibited. This is primarily a way to use UDP traffic on Ethernet for security purposes. Prevents data from leaving the local network, regardless of the IP address used for the destination address	Boolean integer
So_keepalive	You can keep packets of TCP traffic contiguous. These packets can make sure that the connection is maintained when no information is transmitted.	Boolean integer
So_oobinline	The abnormal data received can be considered as normal data, which means that the data will be received through a standard call to recv ().	Boolean integer
So_reuseaddr	When the socket is closed, the port number that the local side uses for the socket can be reused immediately. In general, it can be reused only after the system has been defined for a period of time.	Boolean integer

A more common usage is that of setsockopt (socket. Sol_socket,socket. so_reuseaddr,1) Here value is set to 1, which means that the SO_REUSEADDR is marked as true, and the operating system releases the port of the server immediately after the server socket is closed or the server process terminates, otherwise the operating system retains the port for a few minutes.

Third, file upload and the breakpoint continued to pass

Here we write an example, to achieve file breakpoint continuation, how to solve the problem of possible sticky, here I just simple simulation under the interruption of operation, the article finally I will give a simple ftp gadget, inside the more perfect to achieve the function of the continuation of the breakpoint. Let's take a quick look at the code

# server.py# Transceiver File Import Socketserver_socket = Socket.socket () server_socket.bind ((' 127.0.0.1 ', 8000,)) Server_ Socket.listen (4) conn, address = Server_socket.accept () toatl_size =int (str (CONN.RECV (1024x768), encoding= ' Utf-8 ')) Conn.sendall (bytes (str (toatl_size), encoding= ' Utf-8 ')) # fix problem with sticky pack have_recv = 0f = open (' 22.txt ', ' WB ') while True:if ha Ve_recv = = Toatl_size:break Else:ret = conn.recv (1024x768) f.write (ret) Have_recv + = Len (ret) F.close () # client.py# send and receive files import socketimport Osclient_socket = Socket.socket () client_socket.connect (' 127.0.0.1 ', 8000,)) File_size = Os.stat (' 1.txt '). St_size # When the client passes the file, the client tells the server file size Client_socket.sendall (bytes (str (file_size), en coding= ' Utf-8 ') # When the hair is first saved in the buffer, may appear sticky packet data = CLIENT_SOCKET.RECV (1024) # Accept the server To the file size of the hair HAVE_RCV = 0if int (str (data, encoding= ' utf-8 ')) = = File_size:with Open (' 1.txt ', ' RB ') as F:for line In F:client_socket.sendall (line) havE_RCV + = Len print (' Intermission, wait for another pass ') break # interrupts after a line is transmitted, analog breakpoint with Ope N (' 1.txt ', ' RB ') as F: # Analog Breakpoint Continuation F.seek (HAVE_RCV) # The file pointer refers to the part for which the upload has been completed Line in F:client_socket.sendall (line) Client_socket.close ()

Iv. IO multiplexing

I/O multiplexing refers to the mechanism by which multiple descriptors can be monitored, and once a descriptor is ready (usually read-ready or write-ready), the program can be notified of the appropriate read and write operations. Let's take a look at Select,poll,epoll's introduction

Select Select, which first appeared in the 4.2BSD in 1983, is used by a select () system to monitor an array of multiple file descriptors, and when select () returns, the ready file descriptor in the array is changed by the kernel to the flag bit. Allows the process to obtain these file descriptors for subsequent read and write operations. Select is currently supported on almost all platforms, and its good cross-platform support is one of its advantages, and in fact it is now one of the few advantages it has left. A disadvantage of select is that the maximum number of file descriptors that a single process can monitor is limited to 1024 on Linux, but can be improved by modifying the macro definition or even recompiling the kernel. In addition, the data structure maintained by select () stores a large number of file descriptors, with the increase in the number of file descriptors, the cost of replication increases linearly. At the same time, because the latency of the network response time makes a large number of TCP connections inactive, but calling select () takes a linear scan of all sockets, so this also wastes some overhead. Poll Poll was born in 1986 in System V Release 3, and it does not differ substantially from select in nature, but poll has no limit on the maximum number of file descriptors. The disadvantage of poll and select is that an array containing a large number of file descriptors is copied in between the user state and the kernel's address space, regardless of whether the file descriptor is ready, and its overhead increases linearly as the number of file descriptors increases. In addition, when select () and poll () file descriptors are ready to tell the process, if the process does not have IO operations on it, the next time you invoke select () and poll (), the file descriptors are reported again, so they generally do not lose the ready message. This approach is called horizontal trigger (level triggered). Epoll until Linux2.6 the implementation method that is directly supported by the kernel is epoll, which has almost all the advantages previously mentioned, and is recognized as the best-performing multi-way I/O readiness notification method under Linux2.6. Epoll can support both horizontal and edge triggering (edge triggered, which only tells the process which file descriptor has just become ready, it only says it again, and if we do not take action then it will not be told again, this way is called edge triggering), The performance of edge triggering is theoretically higher, but the code implementation is quite complex. Epoll also only informs those file descriptors that are ready, and when we call Epoll_wait () to get the ready file descriptor, the return is not the actual descriptor, but a value that represents the number of ready descriptors, and you only need to go to epoll the specifiedIn order to get the corresponding number of file descriptors in the array, the memory mapping (MMAP) technique is used, which eliminates the cost of copying these file descriptors at system call. Another essential improvement is the epoll adoption of event-based readiness notification methods. In Select/poll, the kernel scans all monitored file descriptors only after a certain method is called, and Epoll registers a file descriptor beforehand with Epoll_ctl (), once it is ready based on a file descriptor, The kernel uses a callback mechanism like callback to quickly activate the file descriptor and be notified when the process calls Epoll_wait ().

Since WinDOS only supports SELECT, let's take Select for example,

Handle list 11, handle list 22, handle list = Select.select (handle sequence 1, handle sequence 2, handle sequence 3, timeout) parameter: Four parameters (first three must) return value: Three list the Select method is used to monitor the file handle, and if the handle changes, Then gets the handle. 1, when the handle in the parameter 1 sequence is readable (ACCETP and read), then gets the changed handle and adds to the return value 1 sequence 2, when the parameter 2 sequence contains a handle, then all the handles in the sequence are added to the return value 2 sequence 3, when the handle in the parameter 3 sequence error occurs, The handle to the error is added to the return value 3 sequence 4, when the time-out is not set, the select will block until the listening handle changes when the time-   out = 1 o'clock, if there is no change in the listening handle, then select will block 1 seconds, then return three empty list, If the handle of the listener is changed, it is executed directly.

# serve.py# Listen to multiple ports with select and implement read/write separation import Selectimport Socketsk = Socket.socket () sk.bind ((' 127.0.0.1 ', 8000,)) Sk.listen (5) inputs = [Sk]outputs = []message_dict = {} # stores information received by each client while True:r_list, w_list, e_list = select.se Lect (inputs, outputs, [], 1) print (len (inputs)) for sk_or_conn r_list:if sk_or_conn = = SK: # Once there Client connection, SK changed conn, addr = Sk.accept () inputs.append (conn) # Add the client's conn to the Listener list mess        Age_dict[conn] = [] # with the client's conn as key, generate a new list to store the received information # Conn.sendall (bytes (' Hello ', encoding= ' utf-8 ')) Else:try:ret = SK_OR_CONN.RECV (1024) # Listener list If there is a client sending message, except Exception as ex : Inputs.remove (Sk_or_conn) # If the client disconnects, remove else:data = str (ret, encodi) from the list of listeners　　　　　　　　 ng= ' Utf-8 ') message_dict[sk_or_conn].append (data) # Put the listening information into the dictionary outputs.append (sk_or_conn)             # put it in the outputs inside #   # Sk_or_conn.sendall (bytes (data+ ' Hello ', encoding= ' Utf-8 ')) for Conn in w_list: # separate write operation message = message_dict[conn][0] Conn.sendall (bytes (message+ ' Hello ', encoding= ' Utf-8 ')) del Message_dict[con N][0] Outputs.remove (conn) #client1. Pyimport socketclient_socket = Socket.socket () client_socket.connect (' 127.0.0.1 ', 8000,)) while TRUE:INP = input (' >>> ') client_socket.sendall (bytes (INP, encoding= ' Utf-8 ')) data = str ( CLIENT_SOCKET.RECV (1024x768), encoding= ' Utf-8 ') print (data) client_socket.close () #client. Py2import Socketclient_socket = Socket.socket () client_socket.connect ((' 127.0.0.1 ', 8000,)) while TRUE:INP = input (' >>> ') client_socket.se Ndall (Bytes (INP, encoding= ' Utf-8 ')) data = str (CLIENT_SOCKET.RECV (1024x768), encoding= ' Utf-8 ') print (data) client_socket. Close ()

Here the socket server is compared to the native socket, and he supports that when a request is no longer sending data, it will not wait but can process other requested data. However, if each request takes longer to take, the server side of the Select version cannot complete the simultaneous operation. and select, the implementation is a pseudo-concurrency.

Wu, Socketserver

The Soket server implemented by Threadingtcpserver creates a " thread " For each client that is used to interact with the client. First look at the inheritance diagram

　　　　　　　　　　　　

if __name__ = = ' __main__ ':    address = (' 127.0.0.1 ', 9999)    server = Socketserver. Threadingtcpserver (address, Mytcphandle)    server.serve_forever ()

The internal invocation process for the above code is:

• Start the service-side program
• Executes the tcpserver.__init__ method, creates the server-side socket object and binds the IP and port
• Executes the baseserver.__init__ method, assigning a custom inherited class Myrequesthandle from Socketserver.baserequesthandler to self. Requesthandlerclass
• Executes the Baseserver.server_forever method, while the loop is always listening for client requests to arrive ...
• When a client connection arrives at the server
• Executes the Threadingmixin.process_request method, creating a "thread" to handle the request
• Execute the Threadingmixin.process_request_thread method
• Executes the Baseserver.finish_request method and executes self. Requesthandlerclass () is the construction method that executes the custom Myrequesthandler (automatically calls the constructor of the base class Baserequesthandler, which in turn calls the Handle method of Myrequesthandler)

In front of a sockeserver article, you can go to see

Vi. Examples of FTP

Because the example below the directory too many, here is not good to upload, inside with a socketserver implementation of the FTP server, as well as the client, support registration login, file breakpoint continuation, and simple command operation, need to see the words can go to https://github.com/Wxtrkbc/ftp look at

　

Python full stack development 10, network programming