What about the firemen? (4)-About flow control

This is a discussion of the flow control, flow control can be very simple, but also can be very complex, especially dynamic flow control. We have a product in the T country a carrier is in trouble, the carrier's parent company is a European operator, and Europe's operators for product acceptance of the harsh is famous, and this time to bring us trouble is the flow of control.

The general function of this product is to send a subscription short message or MMS content to the mobile phone, similar to a few years ago fire for a period of time mobile newspaper. The flow control of this product is somewhat complex:

1, the first of the system deployment is a cluster, assuming that the cluster has 6 hosts, this 6 host of the message volume per second can not exceed a value, assuming 10000, why this requirement, is because the downstream execution of the message to send a short message gateway or MMS Gateway has flow control, you send the fast, the downstream system will reject you, So the total TPS is not super.

2, the source of these messages are from an SP or CP, each CP or SP on the signing platform, it has a maximum number of transmission per second, sent into the volume of MMS, SMS and WAP push channel.

3, in addition to the above limitations, CP and SP are also prioritized, if the low and high priority when sent together, must wait until the high priority first sent to complete.

There are some rules that require, time is longer, I remember not particularly clear.

This system in domestic and abroad many places are on-line, can not say the quality is good, also not too big problem, I take over this product about a few months time in this ringtones encountered a problem, customer complains they found 2 serious problem through the actual monitor of the line, if not solve, they will ******:

1, the system during the peak of the actual amount of traffic per second did not reach the traffic they purchased at that time, such as they buy 10,000/second, but they actually monitor the peak time, there may be only 9,000/second.

2, the system's flow control is not stable, when the contract signed flow control fluctuation can not be greater than plus or minus 10%. For example, suppose they have one of the largest SP, the exclusive sending of messages at 10 o'clock in the morning to 11, they are set to send by 200 per second, when there is no other SP preemption channel, then the message sending rate should be at 180~220 Bar/sec, but the actual rate of transmission will be outside this interval.

At that time, I took over the product for several months, the system of the general realization of understanding, but the details are not clear, responsible for this product team leader find me to help, do not understand also to bite the bullet. At first thought is a small problem, the product development backbone tried several programs over a week, found that it is difficult to meet customer requirements, then the customer and the project manager lost Patience, began to put the problem to senior leadership report, once the senior leaders know, the consequences can be imagined. I was with the product of the team leader and development backbone began to do this problem, and before and after a total of 3 weeks of time (half of the time is basically all overnight).

After a few days of day code and actual test verification, I found that the original flow control implementation of a serious flaw, the original scheme is the storage process to achieve the above flow control of the allocation of traffic, the stored procedure runs every 1 seconds, each time it will be issued by the current task according to the total flow control, The CP and SP are calculated once, then load the required data into the Oracle partition table, insert the rate to be sent into the task table, then the cluster is each task issued by the thread from the task table read to the task, and then loaded from the partition table on the task to send data, The cycle of stored procedures continuously calculates the allocation tasks, and the cluster loads task data for each node to send data.

The initial development of this product in about 2004-2005 years, the original designers can not find, but I guess why the database to solve, because how to control the messages sent per second in the cluster is not a good implementation, and database storage centralized data is the simplest way to implement. This scheme for most operators of the accuracy requirements of the convection control is not so high, in fact, is not too big a problem, as long as the system load is not too heavy, the message can be basically accurate transmission.

Back to the customer found 2 questions, can be roughly perceptual analysis of the reasons:

1, because the assignment of the task is recalculated every second, after the completion of the calculation, issued by the thread to load the data from the partition table, which will take time, that is, the data is not immediately issued, will cause the rate of execution of the previous allocation is stretched, so it is difficult to reach the total capacity

2, Flow control inaccuracy: Single thread sent by the flow control algorithm optimization problem, I will discuss this problem later

For the first problem, we finally designed a flow control center of the application, abandoned the original through the database to assign the task of the logic, all other applications to send messages through the Netty connected to the flow control center, the current task at what rate is sent completely by the flow control center of the indicator, In the Flow control center, its computational logic is fully implemented in memory. Flow control Center and messaging applications between the two-way communication, when the rate to adjust, the flow control center can be the rate of active delivery to the message center. Because of the time tight, we did not use a dedicated machine to deploy the flow control center, but instead of all the sending message host at the start of the time to write its own IP address into a table in the database, and then in this table first inserted data of the machine is compatible to flow control, by the way to send a message. Because flow control is not very performance-intensive, even sending messages has little impact on performance. If the machine that is responsible for the flow control is hung up, then the heartbeat mechanism will remove the machine that hangs up, and then the other node that sends the message will reconnect to the new host that is responsible for the flow control.

See this, we may think this move is very earth, actually at that time I wanted to use Jgroup to do cluster management, but if you used jgroup in production environment, you will find this thing is too complex, I was in a lot of projects have been pits. At that time, we did not see any better open source components of cluster communication, and with the project time tight, we implemented it in a more secure scenario.

For the second problem, the difficulty is relatively small. The original scheme uses a timer, the timer runs once per second, each time the operation of a semaphore to be sent to the TPS, each send thread before sending Accquire, if it can be taken to send, not to be blocked, And then the next 1 seconds to release this semaphore to the sending TPs. This implementation will result in the delivery of a non-average within 1 seconds, such as: I want to send 30 messages in a second, it is possible in 1/3 seconds when the message has been sent out, and then in the remaining 2/3 seconds nothing, so if the downstream sampling statistics TPS is not calculated by 1 seconds, but by 1/6 calculation, The apparent transmission rate is not stable.

Improved flow control algorithm is a reference to a brother product scheme, called sliding window flow control algorithm, because the drawing compared to spend time, I briefly describe. This algorithm will be divided into 10 small windows 1 seconds, or the above to send 30 per second as an example, in the first small window, I need to send 3 messages, to the 2nd small window I only need to accumulate to send 2*3=6 message, to the 3rd small window, I only need to send 3*3=9 messages, The benefit of this algorithm is that the message rate sent is smoother, smoothing out the fluctuations in the rate at which the downstream is sent. This algorithm also has a benefit, unlike the above algorithm to have a timer constantly zeroing, and only need a very simple to get the current system time-system startup time, you can calculate the current in which the sending window, the system is not only the cost of small and more accurate.

After the above 2 optimization, to achieve the desired target, the following baseline version of the product flow control algorithm has been re-designed, and later how to optimize the essay. After 1 years of baseline version optimization, the product of the team leader left the company, the product's development backbone to the overseas resident, there is no contact. But when I wrote this, I thought 3 of them were like a ghost in the company. Discussion plan, change code, test ...

What happened to the Fireman (4)-About flow control