I learned how to use Python Socket to develop concurrent FTP servers in one hour !!

I learned how to use Python Socket to develop concurrent FTP servers in one hour !!

 

What is socket?

What is socket? socket is also called socket. It is used to describe IP addresses and ports and is a communication chain handle. Applications usually send requests to or respond to network requests through sockets. To put it bluntly, it is a communication mechanism. It is similar to the telephone customer service departments of banks and China Telecom. When you make a call, there will be someone there to answer your questions. The customer service department is equivalent to the socket server, and your side is equivalent to the client. Before the call ends, it is impossible for someone to talk to you, because you are communicating with him. Of course, the telephone switches of the customer service department will not be allocated repeatedly. The httpsmtptp and other network protocols we use every day are implemented based on the upper layer of the socket. No matter what network protocols we use, they are essentially receiving and sending data, however, the data type and content to be sent are different. The "send" and "receive" actions are the main way for the socket to process data.

Socket originated from Unix, and one of the basic philosophies of Unix/Linux is "Everything is a file". You can use the "open-> read/write-> close" mode to operate. Socket is an implementation of this mode. socket is a special file, and some socket functions are operations (read/write IO, open, and close) on it ), the socket module of python is a unix socket library directly called. Let's take a look at how to implement socket in python.

 

When using socket, you must specify the Socket Family (address cluster), which includes the following:

Socket. AF_UNIX can only be used for inter-process communication in a single Unix System

Socket. AF_INET is used for network communication between hosts.

Socket. AF_INET6 IPv6 communication

To implement communication between hosts, we need to use socket. AF_INET.

 

After confirming the address cluster, you also need to specify the socket Data Type

Socket. SOCK_STREAM stream socket, for TCP

Socket. SOCK_DGRAM datagram socket, for UDP

Socket. the original socket of SOCK_RAW. A common socket cannot process network packets such as ICMP and IGMP, while SOCK_RAW can. Secondly, SOCK_RAW can also process special IPv4 packets. In addition, the original socket, you can use the IP_HDRINCL socket option to construct an IP address header.

Socket. SOCK_RDM is a reliable UDP format, that is, ensure the delivery of datagram but not the order. SOCK_RAM is used to provide low-level access to the original protocol. It is used when special operations are required, such as sending ICMP packets. SOCK_RAM is generally only used by programs run by senior users or administrators.

Socket. SOCK_SEQPACKET reliable continuous packet Service

 

We usually use SOCK_STREAM (for TCP) and SOCK_DGRAM (for UDP ).

 

 

Functions that may be used during socket call:

S = socket (family, type [, protocal]) uses the given address family, socket type, and Protocol Number (0 by default) to create a socket.

 

The socket instance has the following methods:

S. bind (address) binds the socket to the address. The address format depends on the address family. In AF_INET, the address is expressed in the form of a tuple (host, port.

S. listen (backlog) starts listening for incoming connections. Backlog specifies the maximum number of connections that the operating system can suspend before a connection is denied. This value must be at least 1, and most applications can be set to 5.

S. connect (address) is connected to the socket at address. Generally, the address format is a tuples (hostname, port). If you connect to a server on the same machine, you can set the hostname to 'localhost '. If a connection error occurs, socket. error is returned.

S. connect_ex (adddress) function is the same as connect (address) function, but 0 is returned for success, and errno is returned for failure.

S. accept () accepts the connection and returns (conn, address). conn is a new socket object and can be used to receive and send data. Address is the address used to connect to the client.

S. close () to close the socket.

S. fileno () returns the file descriptor of the socket.

S. getpeername () returns the remote address of the connection socket. The returned value is usually a tuples (ipaddr, port ).

S. getsockname () returns the address of the socket itself. It is usually a tuple (ipaddr, port)

S. getsockopt (level, optname [. buflen]) returns the value of the socket option.

S. gettimeout () returns the value of the current superperiod in seconds. If no superperiod is set, None is returned.

S. recv (bufsize [, flag]) accepts socket data. The data is returned as a string. bufsize specifies the maximum data size to receive. Flag provides other information about messages, which can be ignored.

S. recvfrom (bufsize [. flag]) is similar to recv (), but the return value is (data, address ). Data is the string containing the received data, and address is the socket address for sending data.

S. send (string [, flag]) sends data in string to the connected socket. The returned value is the number of bytes to be sent, which may be smaller than the size of the string.

S. sendall (string [, flag]) sends data in string to the connected socket, but will try to send all data before returning. If None is returned successfully, an exception is thrown.

S. sendto (string [, flag], address) sends data to the socket. address is a tuples in the form of (ipaddr, port), specifying a remote address. The returned value is the number of bytes sent. This function is mainly used for UDP.

S. setblocking (flag) if the flag is 0, the socket is set to non-blocking mode; otherwise, the socket is set to blocking mode (default ). In non-blocking mode, if no data is found when recv () is called, or the send () call cannot send data immediately, a socket. error exception occurs.

S. setsockopt (level, optname, value) sets the value of the given socket option.

S. settimeout (timeout) sets the superperiod of socket operations. timeout is a floating point number, measured in seconds. If the value is None, there is no superperiod. Generally, it should be set when the socket is just created during the superperiod, because they may be used for connection operations (such as connect () Ordinary non-socket Instance functions.

Getdefatimetimeout () returns the default socket timeout (in seconds ). None indicates that no timeout is set.

Gethostbyname (hostname) converts a Host Name (for example, "www.baidu.com") to an IPv4 address. The IP address is returned as a string, for example, "8.8.8.8.8 ". IPv6 is not supported

Gethostname () returns the Host Name of the local machine.

 

 

Code officially written !!!

 

Below is the simplest socket communication:

 

Server. py server # EchoserverprogramimportsocketHOST = ''# null represents 0.0.0.0PORT = 50007 # listening port s = socket. socket (socket. AF_INET, socket. SOCK_STREAM) # generate a ettcp communication instance, s. bind (HOST, PORT) # bind the ip address and PORT. Note that bind only accepts one parameter and (HOST, PORT) is made into a ancestor and transmitted to s. listen (1) # Start listening. The number in it indicates the maximum number of connections that can be suspended by the server before the new connection is rejected. However, if the experiment is over, you can write 1 to conn, addr = s. accept () # accept the connection and return two variables. conn indicates that after each new connection enters, the server generates a new instance, which can be used for sending and receiving, addr is the address of the connected client. The accept () method returns conn and addr when a new connection is in. If there is no connection, this method will block until there is a new connection. Print 'connectdby', addrwhileTrue: data = conn. recv (1024) # receive 1024 bytes of data ifnotdata: break # If the client data cannot be received (indicating that the client is disconnected), disconnect the conn. sendall (data. upper () # convert all received data into uppercase and send it to the client conn. close () # close the connection

 

Client. py client importsocketHOST = '2017. 168.3.1 '# remote socket server ipPORT = 50007 # remote socket server port s = socket. socket (socket. AF_INET, socket. SOCK_STREAM) # instantiate sockets. connect (HOST, PORT) # connect to the socket server whileTrue: msg = raw_input (Yourmsg ::). strip () # Let the user enter the message, remove the carriage return and space iflen (msg) = 0: continues. sendall (msg) # send the message data = s to the server. recv (1024) # print 'Received ed', datas. close ()

 

In this way, we can implement one-to-one communication between the server and the client, but you will find that the server is also interrupted as long as the client is interrupted in a middle, which is obviously unreasonable, if a client is disconnected, the server must be able to continue providing services to other clients. Why will the server end be closed along with the client, because the server code

ifnotdata:break

If the client data is not received, it will jump out of the loop. We can adjust the representative to the following:

# EchoserverprogramimportsocketHOST = ''# null represents 0.0.0.0PORT = 50007 # listening port s = socket. socket (socket. AF_INET, socket. SOCK_STREAM) s. bind (HOST, PORT) s. listen (1) whileTrue: conn, addr = s. accept () print 'connectdby', addrwhileTrue: data = conn. recv (1024) # receive 1024 bytes of data ifnotdata: break # If the client data cannot be received (indicating that the client is disconnected), disconnect the conn. sendall (data. upper () # convert all received data into uppercase and send it to the client conn. close () # close the connection instance of this client

In this way, if a client is disconnected, the loop in the innermost part will jump out and return to the while loop at the first layer,

conn,addr=s.accept()

The above accept () method will continue to wait for a new connection, so that the server can continue to provide services for the client.

 

After the problem of service continuity is solved, a new problem arises. When you start the server and start two more clients at the same time, you will find that, only one client can continuously communicate with the server, and the other client will remain suspended. When you disconnect a client that can communicate, you can find that 2nd clients can communicate with the server. Why? Haha, because your server can only provide services to one customer at the same time. Just like when you talk to a person, you cannot talk to others at the same time, right?

 

In fact, it is easy to allow your service port to communicate with multiple clients at the same time. Simply use multi-thread concurrency. What? You won't write multithreading? It doesn't matter. Python has already been implemented for you. You just need to call a module named SocketServer. Next we will change the single-thread socket server to multi-thread:

 

Importetserverclassmytcphandler (SocketServer. baseRequestHandler): # inherit the BaseRequestHandler base class, and then rewrite the handle method, and implement all interactions with the client in the handle method. defhandle (self): whileTrue: data = self. request. recv (1024) # receive 1024 bytes of data ifnotdata: breakself. request. sendall (data. upper () if _ name __=__ main __: HOST, PORT = localhost, 50007 # pass the class just written as a parameter to the ThreadingTCPServer class, the following code creates a multi-thread socketserverserver = SocketServer. threadingTCPServer (HOST, PORT), MyTCPHandler) # start this server, which will always run unless you press ctrl-C to stop the server. serve_forever ()

 

Now, start several more clients to see if all clients can communicate with the server at the same time.

 

Next, let's take a look at how to simulate an ftp server through socket, allowing the client to upload and download files.

 

FTP Client

# _ * _ Coding: UTF-8 _ * ___ author __= 'jile' importetimportosclassftpclient (object): def _ init _ (self, host, port): self. sock = socket. socket (socket. AF_INET, socket. SOCK_STREAM) self. sock. connect (host, port) # connect to the server defstart (self): # Call this method to start the client self after instantiating the client class. interactive () # interaction with users in this method definteractive (self): whileTrue: user_input = raw_input (>> :). strip () iflen (user_input) = 0: continueuser_input = user_input.split () # The user input command is split. The first parameter indicates the action to perform, such as getremote_filenameifhasattr (self, user_input [0]): # determine whether the class has the get or other input method func = getattr (self, user_input [0]) # obtain the Memory Object func (user_input) of the corresponding method in the class through the string # Call this memory object else: print [31; 1 mwrong1_usage [0 mdefget (self, msg ): # download the file print '-- getfunc ---' from the server, msgiflen (msg) = 2: file_name = msg [1] instruction = FileTransfer | get | % s % file_name # Tell the server what file self is to be downloaded. sock. send (instruction) feedback = self. sock. recv (100) # Wait for the server to confirm the message print '-->', feedbackiffeedback. startswith (FileTransfer | get | ready): # indicates that the file on the server exists, and the server is ready to send the file to the client file_size = int (feedback. split (|) [-1]) # in the confirmation message sent back from the server, the last value is the file size. You must know the file size to know the total amount of self. sock. send (FileTransfer | get | recv_ready) # Tell the server that it is ready to receive recv_size = 0 # because the file may be large and cannot be received at a time, it must be collected cyclically every time it is received, count f = file ('client _ recv/% s' % OS. path. basename (file_name), 'wb ') # create a new file locally to store the file content to be downloaded. print' ---> ', file_namewhilenotfile_size = recv_size: # As long as the total size of the file and the size of the received do not want to wait, it means that the iffile_size-recv_size is confiscated> 1024: # The total size of the file minus the size of the received is equal to the size of the remaining not received, if the number is greater than 1024, it indicates that the data cannot be collected at a time, and data = self will be iterated several times. sock. recv (1024) #1024 bytes are received this time, but the actually received data may be smaller than 1024. Therefore, the actual number of received data must prevail. recv_size + = len (data) # received size plus the actual size received in this cycle else: # If the last remaining is less than 1024, all the remaining data will be collected at one time. data = self. sock. recv (file_size-recv_size) # recv_size = file_size # It cannot be written this way, because this time may not be able to receive it all at once, because the actual received data may be less than the data you specified, so we need to write recv_size + = (file_size-recv_size) f. write (data) # write the received content to the file printfile_size, recv_sizeelse: print '--- recvfile: % s ---' % file_namef.close () else: printfeedbackelse: print [31; 1 mwrong0000usage [0 mdefput (self): passdefls (self): passdefcd (self): passdefdelete (self): passif _ name __=__ main __: f = FtpClient ("localhost", 9002) f. start ()

 

 

Server

# _ * _ Coding: UTF-8 _ * _ importetserverimportosclassmytcphandler (SocketServer. baseRequestHandler): defhandle (self): whileTrue: instruction = self. request. recv (1, 1024 ). strip () # receive the Client Command ifnotinstruction: breakinstruction = instruction. split ('|') # split the messages sent from the client by |. The message format is "FileTransfer | get | file_name" ifhasattr (self, instruction [0]). # determine whether the class has this method func = getattr (self, instruction [0]) # obtain the Memory Object func (instruction) of this method # Call this method defFileTransfer (self, msg): # send and receive the file print '--- filetransfer ---', msgifmsg [1] = 'get': # If the command sent by the client is get, that is, download the printclientwantstodownloadfile:, msg [2] ifos. path. isfile (msg [2]): # determine whether the file name sent by the client exists and is a file file_size = OS. path. getsize (msg [2]) # Get the file size res = ready | % s % file_size # tell the client else the file size: res = filedoesn' texist # The file may not exist. send_confirmation = FileTransfer | get | % s % resself. request. send (send_confirmation) # send a confirmation message to the client feedback = self. request. recv (100) # wait for confirmation from the client. If the client sends the file content to the client immediately after confirmation, the socket sends and receives the file to reduce IO operations, when the buffer is full, the message will be sent. The previous message may be merged with a part of the file content into a message and sent to the client. This will become a sticky package, therefore, after waiting for a confirmation message from the client, the two messages are sent separately. The packet iffeedback = 'filetransfer | get | recv_ready 'is no longer attached ': # If the client says it is ready to receive f = file (msg [2], 'rb ') send_size = 0 # the logic of sending is the same as the logic of client-side cyclic receipt whilenotfile_size = send_size: iffile_size-send_size> 1024: data = f. read (1024) send_size + = 1024 else: # leftdatalessthan1024data = f. read (file_size-send_size) send_size + = (file_size-send_size) self. request. send (data) printfile_size, send_sizeelse: print '--- sendfile: % sdone ----' % msg [2] f. close () elifmsg [1] = 'put': passif _ name __= = '_ main _': HOST, PORT =, 9002 server = SocketServer. threadingTCPServer (HOST, PORT), MyTCPHandler) server. serve_forever ()

 

Now, the client can download files from the server! Of course, you only need to download the file now, but the business logic for uploading and viewing the file list is basically the same as this. You can expand it yourself. Of course, if you really want to simulate a more comprehensive ftp function, you have to add user authentication and permission authentication. You can switch directories, set the user's upload space quota, and allow uploading directories, more detailed functions such as multi-user concurrency can be implemented. I have an example in git that includes user authentication, which limits users to be able to operate only in their home directories and allow users to switch directories independently, you can write it by yourself. If you have no idea, refer to my code.

 

 

 

* Note: Some people ask, how many concurrent users can this multi-thread support? The answer is not much support. Of course, I have not actually tested many of them. It is estimated that there will not be more than a few hundred, because the multi-thread of Python cannot take advantage of multiple cores, therefore, we can see that concurrency is actually serialized, not just because the cpu continuously switches between different threads, but because GIL needs to ensure thread security, no matter how many threads you generate, Python GIL allows only one thread to run at a time. If you really want to implement multi-concurrency, it would be a good way to adopt asynchronous mode. I will share with you how to use Asynchronous SelectEpoll to implement actual Socket concurrency!