File descriptor and Server programming descriptor in Server programming

File descriptor and Server programming descriptor in Server programming

In linux, everything is a file, and all underlying layers are blocked through the Virtual File System (VFS) mechanism. Users can perform operations on different drivers through a unified interface, each file requires a reference to indicate that the file descriptor is applied at this time. The file descriptor is similar to the handle in widows. Most operations on the file are performed through this descriptor, such as read, write. For each file descriptor, the kernel uses three data structures for management.

(1) Each process has a record item in the Process Table. Each record item contains an open file descriptor table, which can be considered as a vector, and each descriptor occupies one item. Associated with each file descriptor is:

(A) file descriptor flag. (Currently, only one file descriptor flag FD_CLOEXEC is defined)

(B) pointer to a file table item.

(2) The kernel maintains a file table for all open files. Each file table item includes:

(A) File status signs (read, write, add write, synchronization, non-blocking, etc ).

(B) Current file displacement. (That is, the value operated by the lseek function)

(C) pointer to the table entry of the file v node.

(3) Each open file (or device) has a v node structure. The v node contains the file type and pointer information of the functions that perform various operations on the file. For most files, the v Node also contains the I node (index node) of the file ). This information is read into the memory from the disk when the file is opened, so all information about the file is quick to use. For example, the I node contains the file owner, the file length, the device where the file is located, and the pointer to the actual data block used by the file on the disk.

 

After three layers of the above File System encapsulation, each layer is responsible for different responsibilities, from top to bottom the first layer is used to identify files, the second layer is used to manage independent process data, and the third layer is used to manage file system metadata, directly associate a file. One advantage of this layered approach is that the upper layer can reuse the lower layer structure. Multiple file descriptor items may point to the same file table item, or multiple file table items may point to the same V node.

If two independent processes open the same file, each process that opens the file will get a file table item, but the V node pointer of the two file table items points to the same V node, this arrangement allows each process to have its own current displacement of the file, and supports different open methods (O_RDONLY, O_WRONLY, ORDWR ).

When a process creates a sub-process through fork, the file descriptors in the sub-process share the same file table item, that is, the file descriptors of the Parent and Child processes point to the same. Generally, after fork, we disable the fd that we do not need. For example, if the parent and child processes communicate through pipe or socketpair, they usually close the end that they do not need to read (or write. Only when no file descriptor references the current file table item can the close operation actually destroy the data structure of the current file table item, which is somewhat similar to the reference counting idea. This is also the difference between the close function and the shutdown function in network programming. The former can be truly disconnected only when the last process using the socket handle is closed, the latter did not discuss directly disconnecting one side of the connection. However, in a multi-threaded environment, because the parent and child threads share the address space, the file descriptor has only one copy, so you cannot close the fd you don't need in the thread, otherwise, other threads that need this fd will also be affected. Because the file descriptors opened in the child process share the same file table item, in some system Server programming, if the preforking model is used (the server derives multiple child processes in advance, when each sub-process listens to listenfd for an accept connection, it will cause a group shock. Multiple Sub-processes derived from the server call accept and are put into sleep, when the first client is connected, all processes are awakened even if only one process gets connected, which results in impaired performance. See UNP P657.

If exec is called after fork, all file descriptors remain open. This can be used to pass certain file descriptors to the exec program. At the same time, the file descriptor flag FD_CLOEXEC is used to disable the option to keep the open file descriptor when exec is disabled.

You can also use dup or fcntl to explicitly copy a file descriptor pointing to the same file table item. Copy the file descriptor to a numeric value through dup2.

Each process has a file descriptor table, which is independent of each other. The file descriptors between the two processes are not directly related. Therefore, file descriptors can be directly transmitted within the process, however, if the process is passed across the process, unix can use sendmsg/recvmsg to transmit a special file descriptor (see section 15.7 of the book UNP ). The first three file descriptors of each process correspond to standard input, standard output, and standard errors respectively. However, the number of file descriptors that a process can open is limited. If Too many file descriptors are opened, the "Too open files" problem may occur. In the network server, when you call accept through listenfd, It is reflected as an EMFILE error. This is mainly because the file descriptor is an important resource of the system, and the system resources are exhausted, the default value of a single process file descriptor is generally 1024, which can be viewed using the ulimit-n command. Of course, you can also increase the number of process file descriptors, but this is a temporary solution, because when processing high-concurrency services, the server resources are limited, it is inevitable that resources are exhausted.

When combined with the epoll horizontal trigger method to listen for lisenfd connections, a large number of socket connections will fill the TCP connection queue if not processed, listenfd will always generate readable events, and the server will be busy waiting, chen Shuo, author of muduo, a C ++ open-source network library, prepares an idle file descriptor in advance. When an EMFILE error occurs, the idle file is closed and a file descriptor quota is obtained, then accept gets a file descriptor connected to the socket, and then immediately closes. In this way, the client is gracefully disconnected, and the idle file is re-opened, and the "pitfall" is filled in, in case of this situation again.

1 // "occupies" one file descriptor 2 3 int idlefd = open ("/dev/null", O_RDONLY | O_CLOEXEC); 4 ............ 5 6 // then handle this error when an EMFILE error occurs. 7 8 peerlen = sizeof (peeraddr); 9 connfd = accept4 (listenfd, (struct sockaddr *) & peeraddr, & peerlen, SOCK_NONBLOCK | SOCK_CLOEXEC); 10 11 if (connfd =-1) 12 {13 if (errno = EMFILE) 14 {15 close (idlefd ); 16 idlefd = accept (listenfd, NULL, NULL); 17 close (idlefd); 18 idlefd = open ("/dev/null", O_RDONLY | O_CLOEXEC); 19 continue; 20} 21 else22 ERR_EXIT ("accept4"); 23}

Because the file descriptor involves a large amount of content, it will only be used as an example and will be updated later ....