Using tcpcopy drainage to do simulated online test

Introduction of tools

Tcpcopy is a distributed online pressure testing tool, can copy the line of traffic to the test machine, real-time analog online environment, to achieve in the program is not online in the case of real time to assume the effect of line flow, the early detection of bugs, increase the confidence online.

Tcpcopy is an open source project by NetEase Technology department in September 2011 and has now been updated to version 0.4.

Compared with the traditional pressure testing tools (such as: Abench), Tcpcopy's biggest advantage lies in its real-time and authenticity, in addition to a small number of packet loss, complete copy of the line flow to the test machine, the real analog line flow changes in the law.

Second, the principle of tcpcopy

1. Process

Now use Nginx as the front end to illustrate the principle of tcpcopy:

On the left side of the image above is the online front-end machine, and the right side is the test front-end machine. The online front-end machine opens the Tcpcopy client (tcpcopy process), tests the front-end machine to open the Tcpcopy server (interception process), and the Nginx service is started on both machines.

Tcpcopy copy once traffic access is as follows:

① a access to the online front-end machine;

The ②socket packet was copied from the IP layer to the tcpcopy process;

③tcpcopy Modify the purpose of the package and the source address, and send the test front-end machine;

④ copy of the packet arrives at the test front-end machine;

⑤ test the Nginx processing access of the front-end machine and return the result;

⑥ return results are intercepted and discarded at the IP layer and returned by the IP header returned by the intercpetion copy;

The ⑦IP header is sent to the tcpcopy process on the online front-end machine.

1. Code Analysis

1 First, at the link layer or the IP layer, before handing the package to the previous layer, the system checks to see if the process created the socket (Af_packet,sock_dgram,...) or socket (Af_inet,sock_raw,...) A socket of the same type (that is, the original socket Sock_raw), and if so, the packet is copied and sent to the socket's buffer. Tcpcopy is the way to replicate access traffic. The above two methods of grasping package, the former work in the data link layer, the latter work at the IP layer. The grab functions used in different versions of Tcpcopy are different, in version 0.3:

int sock = socket (af_packet,sock_raw,htons (ETH_P_IP));

In version 0.4, you use the following:

int sock = socket (AF_INET,SOCK_RAW,IPPROTO_TCP);

The above two functions work at the link layer and IP layer respectively, the former will crawl in and out of the package, the latter only crawl into the bag.

2 tcpcopy When sending a copy of the packet, the following socket is used:

Sock = socket (af_inet, sock_raw,ipproto_raw);

and set the IP_HDRINCL for this socket:

setsockopt (sock, Ipproto_ip, Ip_hdrincl, &n, sizeof (n));

Therefore, the network layer will no longer increase the IP header. The destination IP and port of the package were changed before sending:

Tcp_header->dest = Remote_port;

IP_HEADER->DADDR = remote_ip;

Finally, call the SendTo function to send the package to the test front-end machine:

Send_len = SendTo (sock, (char *) ip_header,tot_len,0,

(struct sockaddr *) &toaddr,sizeof (TOADDR));

3 Load the Ip_queue module on the test front-end machine and set the iptables rule:

Iptables-i output-p tcp–sport 80-j QUEUE

The replicated access traffic arrives at the Nginx,nginx processing on the test front-end machine and returns the result, which is sent to the target QUEUE at the IP layer by the iptables rule that is previously set. And the queue is implemented by the Ip_queue module. Next, the matching package will be processed by the kernel via NetLink socket to user space (in this is the Tcpcopy server interception process).

The NetLink socket is a communication mechanism between the kernel and the user process, the most common interface for network applications to communicate with the kernel, and can be used to configure various aspects of the network (such as packet filtering).

Interception creates a NetLink socket in the following way:

int sock = socket (af_netlink,sock_raw,netlink_firewall);

The Netlink_firewall protocol has three types of messages: Ipqm_mode,ipqm_packet,ipqm_verdict.

The kernel sends the returned result packet just intercepted via a ipqm_packet message to Interception,interception to send a ipqm_verdict message to the kernel telling the kernel the result of the decision on the packet (drop,accept,etc.). Tcpcopy This method to intercept the results returned by Nginx on the test front-end machine and return an IP header by interception. The corresponding code implementation is as follows:

Copy the IP header of the resulting package, send:

1 2 3 4 5 6 7 8 9 10 11 12-13

struct Receiver_msg_st msg;   ... memset (&msg, 0, sizeof (struct receiver_msg_st));   memcpy (void *) & (Msg.ip_header), ip_header,sizeof (struct IPHDR));   memcpy (void *) & (Msg.tcp_header), tcp_header,sizeof (struct TCPHDR)); ... send (sock, (const void *) msg,sizeof (struct Receiver_msg_st), 0);

Interception send Ipqm_verdict message to kernel report verdict result:

1 2 3