Several questions and tests on TCP concurrent connections

Giantchen at gmail)

Blog.csdn.net/solstice

A few days ago I posted two questions about TCP on Sina Weibo, triggering a discussion of http://weibo.com/1701018393/eCuxDrta0Nn.

The first elementary question is:

There is a machine with an IP address and a TCP service is running on it.ProgramThe program only listens on one port. Question: theoretically (only the layer of TCP/IP is considered,IPv6 is not considered) How many concurrent TCP connections does this service program support? Answer: 65536.

Specifically, this question is equivalent to: the address of a TCP Service Program is 1.2.3.4: 8765. How many concurrent connections can it accept theoretically?

The second advanced question is:

A tested machine A has the same functions as above. There is also machine B on the same switch. If the program B allows direct sending and receiving Ethernet frames, ask: how many resources does B need to assume 0.1 million concurrent TCP connections to? What about the 1 million?

According to the results of the discussion, many people have made the first question, while the second question is almost unattended.

The answer is not published here (see the end of the article for the answer to the first question). Let's continue to think about the essential question: How much system resources a TCP connection will occupy.

In the current Linux operating system, if you use socket ()/connect () or accept () to create a TCP connection, each connection occupies at least one file descriptor (file descriptor ). Why do we say "at least "? Because file descriptors can be copied, such as DUP () or inherited, such as fork (); in this case, multiple file descriptors correspond to the same TCP connection in the system. According to this, many people give the first answer: the number of concurrent connections is limited by the maximum number of files that the system can open at the same time. This answer is correct in practice, but does not match the original question.

If we leave the operating system aside and only consider the TCP/IP layer, what is the overhead for establishing a TCP connection? What is the minimum overhead theoretically? Two scenarios are considered:

1. Suppose there is a TCP Service program. What do I need to do to successfully initiate a connection to this program? In other words, how can we make this TCP Service Program think that a customer is connected to it (so that its accept () call returns normally )?

2. Suppose there is a tcp client program. What do I need to do if the program successfully establishes a connection to the server? In other words, how can I make the tcp client think that it is already connected to the server (so that its connect () call returns normally )?

The above two questions are not about programming, how to call the Sockets API, but how to make the TCP/IP protocol stack of the operating system think that the task has been successfully completed and the connection has been successfully established.

Those who have learned the TCP/IP protocol and understood the three-way handshake understand that the TCP connection is a virtual connection, not a circuit connection, theoretically, maintaining a TCP connection does not occupy Network Resources (system resources of two programs are occupied ). As long as both parties believe that the TCP connection exists and IP packet can be sent to each other, the TCP connection will always exist.

For Question 1To initiate a connection to a TCP Service Program, the client (for the sake of understanding, hereinafter referred to as the faketcp client) only needs to do three things (three-way handshake ):

1A. Send an IP packet to the TCP Service Program, containing the syn tcp segment

1b. Wait for the other party to return a TCP segment containing SYN and ack.

1c. Send a segment containing ack to the recipient

After completing these three tasks, the TCP server program considers that the connection has been established. These three tasks do not occupy client resources (?), If the faketcp client program can bypass the operating system's TCP/IP protocol stack, it can directly send and receive IP packet or Ethernet frame. In other words, the faketcp client can repeat these three events. Each time a different IP: port is used, countless TCP connections are created on the server, and the faketcp client is lossless. Soon we will see how to implement this with a program.

For Question 2To make a tcp client program think that the connection has been established, the faketcp server only needs to do two things:

2a. Wait for the syn tcp segment sent from the client

2b. Send a TCP segment containing SYN and ACK

2c. Ignore the ACK segment sent by the other party

After completing these two tasks (receiving a SYN and sending a SYN + ACK), the tcp client considers that the connection has been established. These three tasks do not occupy the resources of the faketcp server (?) In other words, the faketcp server can repeat these two events and accept countless TCP connections, while the faketcp server itself is lossless. Soon we will see how to implement this with a program.

Based on the analysis of the above two problems, it is meaningless to talk about "TCP concurrent connections" separately, because the number of connections is basically the same. A more meaningful performance indicator may be: "How many messages are sent and received per second", "How many bytes of data are sent and received per second", and "How many concurrent customers are supported.

Faketcp Program Implementation

CodeSee: https://github.com/chenshuo/recipes/tree/master/faketcp can be compiled directly with make

To verify my above statement, I have written several small programs to implement faketcp. These programs can initiate or accept countless TCP concurrent connections without consuming operating system resources, dynamic memory allocation is not used.

I have a PC running Ubuntu Linux 10.04 and the hostname is atom. All the tests are carried out here.

The network configuration in the home test environment is:

In his "talking about network programming Learning Experience", Chen Shuo mentioned that "the tun/TAP device can be used to implement a TCP/IP protocol stack capable of point-to-point communication with the local machine in user mode ", this method can be used in this test.

The network configuration for the test is as follows:

The specific method is: on Atom, create a tun0 virtual network card by opening the/dev/NET/TUN device, and then set the network card address to 192.168.0.1/24, in this way, the faketcp program assumes all the machines in the network segment 192.168.0.0/24. Atom is sent to 192.168.0.2 ~ The IP packet of 192.168.0.254 will be sent to the faketcp program. The faketcp program can simulate any of the IP addresses to send the IP packet to atom.

The program is implemented in several steps.

Step 1: Implement the ICMP echo protocol to ping faketcp.

Code see https://github.com/chenshuo/recipes/blob/master/faketcp/icmpecho.cc

The function that responds to the ICMP echo request is also used in subsequent programs in the https://github.com/chenshuo/recipes/blob/master/faketcp/faketcp.cc#L57 function.

Run the command to open three command line windows:

1. Run sudo./icmpecho in the 1st windows. The program displays

Allocted tunnel InterfaceTun0

2. Run in the 2nd windows

$ Sudo ifconfig tun0 192.168.0.1/24

$ Sudo tcpdump-I tun0

3. Run in the 3rd windows

$ Ping 192.168.0.2

$ Ping 192.168.0.3

$ Ping 192.168.0.234

It is found that each IP address of 192.168.0.x can be pinged.

Step 2: The TCP connection is denied, that is, the RST segment is sent when the syn tcp segment is received.

Code see https://github.com/chenshuo/recipes/blob/master/faketcp/rejectall.cc

Run method: open three command line windows. The operations in the first two windows are the same as those in the front. The faketcp program is./rejectall.

3. Run in the 3rd windows

$ NC 192.168.0.2 2000

$ NC 192.168.0.2 3333

$ NC 192.168.0.7 5555

It is found that all TCP connections initiated to any of the IP addresses are rejected.

Step 3: Implements the function of accepting TCP connections, that is, sending back SYN + ACK when receiving syn tcp segment. This program also handles connection disconnections, that is, sending back the FIN + ACK when receiving the fin segment.

Code see https://github.com/chenshuo/recipes/blob/master/faketcp/acceptall.cc

Run method: Open the three command line windows. The steps are the same as before. The faketcp program is./acceptall. This time, we will find that the NC can be connected to every port of every IP address in 192.168.0.x. You can also run netstat-TPN in the 4th window to confirm that the connection is established. If data is input in NC, the data is accumulated in the operating system, as the length of the sending Queue (send-q) displayed by netstat increases.

Step 4: Receives data based on the TCP connection in step 3, that is, Ack is returned when the TCP segment containing payload data is received.

Code see https://github.com/chenshuo/recipes/blob/master/faketcp/discardall.cc

Run method: Open the three command line windows. The steps are the same as before. The faketcp program is./acceptall. This time, we will find that the NC can be connected to every port in 192.168.0.x, and the data can also be sent out. You can also run netstat-TPN in the 4th windows to confirm that the connection is established and the sending queue length is 0.

This step has solved problem 2 and assumed any TCP server.

  

Step 5: Solve the previous problem 1. Assume that the client initiates any number of connections to atom.

Code see https://github.com/chenshuo/recipes/blob/master/faketcp/connectmany.cc

Different from the preceding method, this step opens four command line windows.

1. Run sudo./connectmany many 192.168.0.1 1st 2007 in the 1000 window, which means that 2007 concurrent connections will be initiated to 192.168.0.1: 1000.

Program display

Allocted tunnel interface tun0
Press the Enter key to start connecting 192.168.0.1: 2007

2. Run in the 2nd windows

$ Sudo ifconfig tun0 192.168.0.1/24

$ Sudo tcpdump-I tun0

3. Run a service program that can receive concurrent TCP connections in the 3rd window, which can be httpd or muduo's echo or discard sample. The program should be port 2007 of listen.

4. Return to the 1st window and press Enter. Then, use netstat-TPN in the 4th window to observe the concurrent connections.

If you are interested, you can continue to expand and perform more tests on TCP to further deepen your understanding and verify that the operating system TCP/IP protocol stack faces different input behaviors. You can even implement a complete TCP state machine and make a simple mini TCP stack as I proposed in "talk about network programming learning experience.

The answer to the first question:

When only IPv4 is considered, the theoretical upper limit of concurrency is 2 ** 48. Considering that some IP segments are retained, the upper bound can be reduced but the order of magnitude remains unchanged. The actual limit is the number of global file descriptors of the operating system and the memory size.

A TCP connection has two end points, each of which is {IP, port}. The question is that one of the end points is fixed, so leave the degree of freedom of the end point, 2 ** 48. The maximum number of client IP addresses is 2 ** 32, and the maximum number of connections initiated by each client IP address is 2 ** 16.

Even if the client uses Nat, this theoretical ceiling is not affected. (Why ?)

In a real Linux system, you can adjust the kernel parameters to support millions of concurrent connections. For details, see:

Http://urbanairship.com/blog/2010/09/29/linux-kernel-tuning-for-c500k/

Http://www.metabrew.com/article/a-million-user-comet-application-with-mochiweb-part-3

(. End .)