HAProxy high concurrency Problem Solving

HAProxy high concurrency Problem Solving

Directory

1. Problem Description
2. Problem Analysis
3 HAProxy Positioning
3.1 connect (...)
3.2 tcpv4_connect_server
3.3 call of tcpv4_connect_server
3.4 call of connect_server
3.4 modify source code to add custom logs
3.5 srv_dynamic_maxconn
4 solutions
5. Email Exchange with the author of HAProxy
5.1 email description
5.2 reply from the recipient
5.3 resend the verification answer
5.4 final response of the other party
6 postscript

Solution to HAProxy current network problems

1. Problem Description

After RMI is launched, the interfaces on the current network always report exceptions.

2. Problem Analysis

Through understanding the RMI source code, this is when there is no available connection on the RMI client side, you need to create a new connection, but the connection fails.

Packet Capture is usually possible to locate network problems, so packet capture finds that the failed connection has a common phenomenon, that is, HAProxy takes the initiative to close the connection in five seconds, considering that the configuration of HAProxy has a connectTimeout parameter of 5 seconds, it should be that HAProxy fails to connect to the back-end RMI server.

Packet Capture also proves this because no SYN packets initiated from HAProxy to the backend (except the handshake caused by check) were found in five seconds ).

Therefore, we suspect that the problem lies in HAProxy. Otherwise, at least HAProxy should initiate an active connection.

In this case, we guess that HAProxy does not obtain the available server.

3 HAProxy Positioning

3.1 connect (...)

I initially suspected that HAProxy didn't get the available server. So where can I solve the problem?

If HAProxy needs to establish a connection to the remote server, it must call the C language API connect (....)

In the Function

You can see that connect (...) is called (...)

3.2 tcpv4_connect_server

View the call stack of tcpv4_connect_server

The above code is in the event_accept function. That is to say, when the client in the session is established, the session server connection function is specified, and the tcpv4_connect_server function is triggered somewhere later.

3.3 call of tcpv4_connect_server

It is clear here that the function is triggered by calling the connect_server function and then based on the previously specified connection function. Because the tcpv4_connect_server function is specified in 3.2, it is triggered, the connect function is called in the tcpv4_connect_server function, so you need to track the connect_server function.

3.4 call of connect_server

View the call stack,

Try to locate the problem through a similar call mechanism.

More detailed call stacks are not listed one by one.

3.4 modify source code to add custom logs

To locate the problem, modify the HAProxy [1.4.23] source code, add your own logs to each session creation and subsequent behavior, and add a unique session ID to each log line.

In this way, you can track the specific behavior of each session.

The log format is as follows:

3.5 srv_dynamic_maxconn

Through the log, we found that it was not because HAProxy could not find the available server, but that the current dynamic maxconn of the server was dynamically computed through this function.

Trace the Code:

Unsigned int srv_dynamic_maxconn (const struct server * s, struct session * session)
{
Unsigned int max;
If (s-> proxy-> beconn> = s-> proxy-> fullconn)
{
/* No fullconn or proxy is full */
Max = s-> maxconn;
}
Else if (s-> minconn = s-> maxconn)
{
/* Static limit */
Max = s-> maxconn;
}
Else
{
Max = MAX (s-> minconn,
S-> proxy-> beconn * s-> maxconn/s-> proxy-> fullconn );
}
If (s-> state & SRV_WARMINGUP )&&
Now. TV _sec <s-> last_change + s-> slowstart &&
Now. TV _sec> = s-> last_change ){
Unsigned int ratio;
Ratio = 100 * (now. TV _sec-s-> last_change)/s-> slowstart;
Max = MAX (1, max * ratio/100 );
}
Return max;
}

Therefore, add logs in this Code section and find that in the ant nest environment, each time this function returns 1.

So the problem is located, and 1 is returned, which leads,

The first request to establish a connection will be responded, and the subsequent 2, 3... Are rejected.

Check the log again to fully verify this point.

4 solutions

Now that we know the problem, how can we solve it?

It must be solved through the logic of this function.

Check the srv_dynamic_maxconn function and find that there are two solutions in configuration.

1. Set minconn as a large parameter.

2. directly set minconn to be the same as maxconn to completely remove the minimum limit. The concurrency is configured according to maxconn.

You can see in the Code for the 2nd Cases

That is, if the two are of the same size, max returns s-> maxconn. This is no problem.

5. Email Exchange with the author of HAProxy

Since it is open-source software, you can directly communicate with the author.

Below is the email exchange with the author.

5.1 email description

5.2 reply from the recipient

5.3 resend the verification answer

So I sent my own answers to view the comments of the other party on our solutions, and did not forget to praise the popularity of the other party's software.

5.4 final response of the other party

That is to say, the author thinks that it is better to directly remove the minconn parameter, so we are in haproxy. this parameter is removed from the configuration of cfg. Through log printing, the value of minconn is equal to that of maxconn, that is, the branch of static limit is adopted.

Now the problem has been solved, and the understanding of HAProxy is better than before.

6 postscript

1. if you encounter a problem, you may encounter difficulties. In particular, you can directly debug or view the source code to solve the problem. Generally, gdb is used for c in linux, and jdb for java can be used for Row-by-row tracking, which is very convenient and accurate!

2. Open-Source Software. There are many pitfalls on the Internet, so you can try to find out if there are solutions.

3. The methodology is very important to the problems solved. This also requires experience. For example, the positioning of HAProxy in this article lies in connect (...) locate the api entry, sort out the call stack, and add logs to gradually locate the problem.

 

Haproxy + Keepalived build Weblogic high-availability server Load balancer Cluster

Keepalived + HAProxy configure high-availability Load Balancing

Haproxy + Keepalived + Apache configuration notes in CentOS 6.3

Haproxy + KeepAlived WEB Cluster on CentOS 6

Haproxy + Keepalived build high-availability Load Balancing

Configure an HTTP Load balancer using HAProxy

For details about HAproxy, click here
HAproxy: click here

This article permanently updates the link address: