Performance optimization for an internal API system

This is a creation in Article, where the information may have evolved or changed.

The purpose of this API system is to solve the problem of large number of concurrent relational database queries, and the main design idea is to cache the data in the table data of PG through the LRU cache algorithm, and request the peak service to reduce the direct database operation and reduce the pressure of the database. Query efficiency is also greatly improved. Since the first edition of the project is not very high, then there are a number of serious problems: including the inability to host millions of terminal-level data, requests for excessive pressure requests fail, memory and CPU are full at peak requests, Goruntine crashes and map read-write conflicts, and performance is far from expected. After the optimization of the project alone had to be a little Dian regression reconstruction, during the encounter a lot of pits ...

This optimization is more of the algorithm and go using techniques and features of optimization, including simplifying the design of recursive algorithm to accelerate the start of data loading, data structure simplification, caching and reuse of data sharing reduce memory pressure, memory and CPU explosion, location and resolution, optimization of GC and memory allocation, channel use pit , the use of timers and so on. I will introduce several real-world scenarios to the important ones.

1. Database Cache data Load optimization at startup

Scenario: When the API system is turned on, it is necessary to quickly load the details of the 100W terminal and the approximately 1-10w unequal packet information from the database, but the speed of the 100W data takes about 1 minutes or more, and the 10W packet takes more than 20 minutes to complete the load.

100W of data want a SQL query down and the cache will definitely have unexpected problems, the amount of data is too large, so that eventually will be stuck. So the use of segmentation strategy, only take 2000 or so at a time, about 50 times, OK, this is no problem. The problem is how to slice and where to start the next query, the original code is as follows:

var MiniD int  shovelsize =  -  if Err := Pg.Get(&MiniD, "SELECT MIN (ID) from client"); Err != Nil {    return Err  }  //Get the smallest ID  Lastsyncedid := MiniD   for {    var Thisqueryresultcount = 0    //Then query the Id>lastsyncedid data    rows, Err := Pg.Queryx(FMT.Sprintf(' SELECT id,mid,gidFrom client WHERE ID >=%d ORDER by id ASC LIMIT%d;        `, Lastsyncedid, shovelsize))    /.../    //lastsyncedid+2000 per time    Lastsyncedid += shovelsize     for rows.Next() {      Thisqueryresultcount++      ...      Err = rows.Structscan(&C)      API.Clientmgr.Replaceinto(&C)      //Lastsyncedid = c.id +1    }    if Thisqueryresultcount == 0 {       Break    }  }

From the above code can see the problem, when the 100W data ID is continuous, each fetch 2000 and then each Lastsyncedid Plus 2000 no problem, once the broken ID, there will be a duplicate query, and insert the cached code to determine whether the current information exists, There is also to overwrite or delete the add, as long as an ID interrupt to repeat about 50 times, 100 interrupts will repeat 50,000 times the delete write operation. So need to modify two places, one is Lastsyncedid = C.id +1, that is, with each query update the last Id, no longer blindly directly increase the 2000, so we guarantee that each check out the information is not the same as before, and the other is to replace replace directly with add , because you can safely insert it without having to judge as many items as before and then insert them. This allows the 100W end data cache to be done in a matter of seconds.

2. Simplify recursive algorithm to improve query efficiency

Scenario: Another factor that affects the speed of initialization is that each time a grouping information is added to the children information of all sub-groups and parent groups, the data is updated and 10W packets are recursively 10W times, causing 10W data to 20+ minutes to load. Get all descendant groups of this group and the packet path from top level to that layer through GID, how to optimize when query gid=1 need 60s+? And once there is a grouping of information changes may need to recursive all the relevant packet cache information, how to do?

Recursive algorithm is an algorithm that directly or indirectly calls its own function or method. The essence is to break the problem down into sub-problems of the same kind of problem, and then recursively invoke the method to represent the solution of the problem. Recursion is a great art that makes the correctness of the program easier to identify without sacrificing performance, but it requires that we can abstract complex problems into the simplest data model to achieve, improper use of not only the efficiency of the greatly reduced, and sometimes unexpected crashes. Before brushing the ACM problem and classmates often say that most of the algorithm can be used to solve the problem of binary tree and recursion, real scene is true.

10W grouped data, each grouping with the ID of the group and the ID of the parent group, as well as the update time of the group information, and other information, the original method of the program is to cache all the packet information, and calculate all of its children ID, Once a group appears modified to roll back the calculation of all children groups and parent groupings, the computational amount is still very large, imagine the group ID if it is 1, the 10W grouping information will be recursively computed. But I have analyzed the following data features to find that the recursive nature of these complex data is unnecessary.

Original Code snippet:

Grouptbl    Map[int]*Rawgroupinfo  type Groupcalcresult struct {      Children   []int      Path       []int      Outputjson []byte  }       func Getallchildrenbyid(ID int, Children []int, Recursivechildrennum int, maxrecursive int) ([]int, int) {    Isend := 0    Recursivechildrennum++    if Recursivechildrennum <= (maxrecursive + 1) {        Allfirstinfo := GM.Grouptbl         for Childrenkey, Childrenvalue := Range Allfirstinfo {            if Childrenvalue.Pid == ID {                Children = Append(Children, Childrenkey)                Children, Isend = Getallchildrenbyid(Childrenkey, Children, Recursivechildrennum, maxrecursive)            }        }    } Else {        Isend = 1    }    return Children, Isend}

You can see that this recursive use is not a problem, in the case of a small amount of data can also be solved, but we can see that there is too much attention, the return value is too much, and these values for our problem itself is not necessary to use, we are concerned about the ID of the parent-child relationship, And the recursive process also has a map read and write conflict, and these read and write is not necessary, nor is the key to our problem. At first I thought it was the recursive algorithm itself problem, I followed the structure of a multi-fork tree expected to solve this problem in algorithm efficiency, after testing I found that the attention of the information is too much and there is no obvious advantage. After careful analysis, in fact, we are only concerned about the ID of the int type and the relationship between the hierarchy of the ID. We abstract this data structure, the 10W grouping, the structure of the following:

1:[2,3,4,5,6,7,8,9]2:[ One, A, -, -, the, -,...] One:[ the,111,...]....

First of all, we can not always maintain this array to preserve the descendants, only need to pay attention to their son, or how to control the coming over, there is no use of the additional information for each grouping in our relationship structure is not necessary, imagine we know a person and their relationship, but also need to wear how many clothes he wears, Did you eat anything today? Two data structures can maintain our grouping information, 10W Group details: Map[gid]gidinfo and group Parent-child information Map[id][]int. After optimization

//Children query for a group IDfunc Childrensearch(groupfamily Map[int][]int, Data []int) []int {  Res :=  Make([]int, 0)   for _, v := Range Data {    Res = Append(Res, v)    Res = Append(Res, Childrensearch(groupfamily, groupfamily[v])...)  }  return Res}

After testing, the 10W packet ID query time changed from the previous 60s+ to about 0.1s.

3.golang Program Memory footprint is not what you see in memory actually occupied

Scenario: Through the GID to obtain all the terminal information, request query gid=1 return data about 32Mb, hundreds of concurrent 16G memory server suddenly occupy the 12g+, how to reduce?

A request to return data 32M, when there are hundreds of such requests when the memory burst 12G, then I was crashing, over and over code, look at Pprof memory data, never found the problem of memory leaks, and later found that This problem is not actually the problem I see the surface cause or the problem is why the memory is not released for a long time, concurrent requests come over the memory burst is actually normal, because after all, a request is 32M, good hundreds of requests at the same time not to increase the blame.

This is actually a pit of Golang memory allocation: In order to ensure the memory in the program is continuous, Golang will request a large chunk of memory (even write only one Hello, world may occupy 100m+ memory). When the user's program requests more memory than was previously requested, the runtime makes a GC and doubles the GC threshold. That is, before the GC is over 4M, the next GC is more than 8M. The reason for our memory explosion is that too much traffic leads to memory requests, and the GC threshold is suddenly larger, the recovery frequency is lower, and the GC predicts that this large memory may be needed later in order to optimize the memory allocation is not released to the system. and Golang adopted a procrastination strategy, even if the memory is freed, runtime will not immediately return the memory to the system, but only if they do not need and the system needs to return to the system. This causes the memory to fall down, a memory leak illusion. So no wonder I've been unable to find the cause of the memory leak.

After finding the cause, the solution is to periodically return the unwanted memory back to the operating system, Debug. Freeosmemory (), but please use caution, preferably not, the best solution is:

1. Minimize object creation

2. Try to do variable reuse, many use common buffer to reduce memory allocation

3. If there are too many local variables, you can put these variables into a large structure, so that when scanning can only scan a variable, to reclaim it contains a lot of memory

4.CPU continues to grow over time

Scene: In the system just debugging completed, the moment the heart is excited to request the correct processing, and then when the service opened, habitual view of the power consumption, memory usage is normal, the CPU in the current increasing, 2%-5%-10%-20%-50%-80%-100%, in just a few minutes on the 100% ... It's a broken heart, just a flow from a bunch of code. How do I quickly navigate to a problem? Where should I start, or what data should I look at to speculate on all the possibilities? Which package is it, which function or even what line of code affects its performance? Here are the main instructions to troubleshoot the positioning process.

1. The first thing to think of in this situation is the case of a dead loop, go back to the code, check if there are no dead loops and other bugs, carefully look at all the possible areas of the cycle, and did not find, and from the symptom is not like a dead loop situation (if it is a dead loop, the CPU growth is not so slow, Must be up at once)

2. Open the GC to see if it is due to the fact that the GC is too much occupied CPU,GC is obviously a 2-minute default, so it is not frequently caused by GC

3. At this time only pprof, view the program's run-time data

Turn on the Pprof interface to view runtime CPU data and heap data.

From the above can be seen flat% the highest is the Rumtime.mach_semaphore_signal method, here only about 28% is actually higher, reached the 60+% (this figure truncated wrong, just, simply, this explains the problem on the line). Ok, know what exactly is causing the CPU high, below we trace the code path, where I use the flame diagram to analyze, more intuitive.

From the flame diagram we can easily analyze the CPU time consumed by each method. It is in SVG format, the mouse can also look at the details. The y-axis is the depth of the stack, and the x-axis is the collection of all the sample points, each of which represents a stack frame. The color is meaningless, just randomly selected. The left and right order is not important. You can look at the widest frame and look up at the bottom, and the fork on the frame represents a different code path. Fast identification and quantification of CPU usage

From the flame diagram information can be seen, in the entire CPU occupied information collection process is almost all runtime information, and is Runtime.timerproc. By analyzing the source of the go language we can know that this is a timer dispatch goruntine. (Code is go1.7.3 source, I have been cut, keep to the effect)

The scheduling process is analyzed as follows:

1. Determine if there is a timer in the heap? If there is no state to set the rescheduling of timers to true, then true means that Timerproc Goroutine is suspended and needs to be re-dispatched. This re-scheduling is the time to add a timer to come in, will be ready to this goroutine. Hang up here Goroutine is using the runtime Park () function.

2. If a timer exists in the heap, take a timer from the top of the heap and determine if it is timed out. After the timeout, delete the timer and execute the method that was mounted in the timer. This step is to loop through the heap until there is no timer in the heap or a timer without a timeout.

3. Before the timer in the heap expires, the goroutine will be in the sleep state, that is, setting the timers sleeping to true. This place is done through the runtime Notesleep () function, and its implementation is dependent on the Futex lock. Here, how long will the Goroutine sleep? It will sleep until the most recent timer expires and starts executing.

From the flame diagram upwards or from the PPROF CPU trend we can see the runtime.mach_semaphore_timewait we mentioned above. So here we can infer that the root of the use of the timer, then the full code to search all the code using the time package method. Found to be and time. Tick () and time. After () two function related. Then the problem is here.

The above two examples, executed alone, will not find any problems, but when you put such code in a 7 * 24-hour service, and the timeout interval is shorter, such as 0.1s, the problem arises.

At first I wondered if there was a dead loop in the code, but after careful inspection of the code, no traces of the dead loop were found, and the algorithm logic was fine. Then restarted the service and found that the CPU usage dropped down. Continue with top observation, bad, this service takes up 1% of the CPU has been rising, after observing a period of time, found that the service on the CPU usage over time to increase.

It can be seen from the flame diagram that almost all Timeproc goroutine are dispatched. Go back to the code and find that there may be problems with only the tick and after.

Back to our code, the Timerproc pointer maintains a goroutine, and the main function of this goroutine is to check if the timer in the small top heap is timed out. Of course, the timeout is to delete the timer and perform the corresponding action of the timer. My timer interval is 1s. So every 1s will create a runtime timer, and through the runtime source, these timer are thrown to the runtime scheduling (a heap). Long time, there will be more than the timer needs runtime scheduling, not consuming CPU is strange.

After finding the reason, continue to analyze the cause of the excessive timer, one is that the time interval in the configuration item is too small (0.1s) one is the code logic bug that was seen above, both of which caused the CPU to soar. Fix bugs in the code, adjust the parameter interval size. The CPU dropped down immediately.

So the question is, how do we use the timer correctly?

Under normal circumstances, the tick represents a permanent alarm clock, as long as you are scheduled to wake you up regularly, after more like a disposable alarm clock, to the time to wake you up, if you did not receive, then it will also stop failure. So the tick doesn't have to be in the loop, or it's going to go on growing. But like the second on the left you find its position is not the innermost loop, so there is no principle of not allowed, to choose the right method in the specific case.

5.channel-Length Traps

Scenario: This is a deep bug that the customer has found, that it is deep, that the bug will be triggered after the request has been received and the database has been interrupted after the normal startup of the program.

One of the rules that I used to stick to when I wrote C + + was that when using arrays, if you want to iterate through the length of the array first, and then traverse it, instead of just the length in the for loop, because once the array changes it can cause unpredictable bugs, but it is usually possible to write directly in the loop. So a lot of people will get used to this notation. The general procedure is OK, but it is best not to do so in Golang, because it is very error-prone to write in concurrency, for example:

if Subchain, OK := Psgrregister[Strings.Join(PSGR, `-`)]; OK {  It should be noted here that you cannot use Len () directly inside the loop, because once the number of close one Chan changes, it can cause serious bugs  Broadcastloop:     for I := 0; I < Len(Subchain); I++ {      Select {       Case Waiter := <-Subchain:        Waiter <- msg        Close(Waiter) //Once the length of close Chan is reduced by 1      default:         Break Broadcastloop      }    }    Close(Subchain)}

You can see that every time you close a Chan, the number of elements in the original Chan array is reduced by 1, so when the concurrent request comes in, it turns out that the last few requests are null, because it ends prematurely here.

6. Use of database triggers with Goruntine

Scenario: During the 60W Terminal Stress test for a bank, there is an explosion in memory consumption because an important function of the API cache system is to monitor the database state and synchronize data changes in a timely manner, we use the form of the trigger to notify the API system, stress testing, Frequent changes and table changes result in a large number of goruntine processing data, up to millions of at peak

This is due to two aspects, one is that our data processing is very slow, probably up to dozens of per second appearance, and then after I optimize the increase of about 10 times times per second to deal with hundreds but did not solve the fundamental problem, another important problem is the design of the problem, that is, two of our business shared a trigger, But one of the only four or five fields in the table is interested, the other modifications do not matter, but we do not separate, resulting in a large number of changes triggered, and our part of the logic of processing performance is not up to, resulting in this problem. The problem was taken into account when it came to the project, but there was no problem at the time, simply ignoring it and successfully validating Murphy's law .

7. Other

The other is a few bits of knowledge points, such as map in concurrent read and write must be locked, slice size initialization is based on, channel do not forget close and so on.

Summarize

From the system optimization of these problems can be seen:

1. When a system is initialized, initialize as fast as possible, such as caching and data preparation, and do not mix initialization and business logic

2. The system architecture design should initially be high specification or higher than the production environment requirements, such as data volume and concurrent request volume standards, or later will be very troublesome, you have a kind of rewriting impulse.

3. Real business scenarios require algorithms that are not as complex as we imagine, most of which can be solved with the most basic recursion, sorting, and search algorithms, and the key is the design of data structures and the abstraction of models.

4. It is important to understand the bottom of a language in any language, not only to give you high-performance code, but also to quickly think of the cause when you encounter problems.

5. In the face of a project first to understand the need to realize the business or function, in writing code to pay attention to the data and algorithmic logic, can not blindly write business code can not extricate oneself

6. Sticking to the most basic code specification will keep you away from a lot of holes