Performance Optimization-Algorithm Optimization

Background

For some reason, our system needs to record the id of a table in another system. However, when the record is complete, other systems may delete some data in the table. In this case, some invalid data is added to the table, we must find these invalid IDs and delete them.

 

At the beginning, our implementation is as follows: we put all the recorded IDs in one list, and then upload them to another system, and return the IDs they have deleted. The specific processing code is as follows:

<Pre name = "code" class = "java"> public String findDeletedStuChooseCourseIds (List <String> stuChooseCourseIds) {List <String> delIds = new ArrayList <String> (); // cyclically traverse all the given IDs and check whether the for (String id: stuChooseCourseIds) {StuChooseCourse stuChooseCourse = commonEao. get (StuChooseCourse. class, id); if (null = stuChooseCourse) {delIds. add (id) ;}return JsonUtils. toJson (delIds );}

At the beginning, the data volume was small, and there was no problem with this method. However, with the passage of time, the amount of data continues to increase, and tens of thousands of data records eventually emerge. The execution time of this method has far exceeded the limit of our patience. Even if the execution is not completed in 5 minutes, this must be optimized.

Analyze the cause of slow execution. It is obvious that it takes time to search for data in the loop. You need to search for data in the database once in each loop, so that you can see it quickly. Therefore, the interaction with the database must be reduced. The code is changed to the following version:

<Pre name = "code" class = "java"> public String findDeletedStuChooseCourseIds (List <String> stuChooseCourseIds) {// obtain all stuchoosecourse idsString nativeSql = "select id from tableName "; list <String> list = commonEao.exe cuteGetNativeSQL (nativeSql); stuChooseCourseIds. removeAll (list); return JsonUtils. toJson (stuChooseCourseIds );}

In this case, you only need to interact with the database once, And the removeAll method of jdk is used (the efficiency of jdk implementation is generally good), the efficiency should be improved a lot. As a result, I perform tests with hope, but the results are still unacceptable, and the efficiency does not seem to have improved much.

The reason for analysis is that stuChooseCourseIds. removeAll (list) is a waste of time, because the list we use is arrayList, and arrayList is time-consuming when performing the search and delete operations. Later, we changed to the upload list, but the results were the same. Therefore, we must change our mindset. The code is changed to the following version:

<Pre name = "code" class = "java"> public String findDeletedStuChooseCourseIds (List <String> stuChooseCourseIds) {List <String> delIds = new ArrayList <String> (); // obtain idsString nativeSql = "select id from tableName" for all stuchoosecourse; List <String> list = commonEao.exe cuteGetNativeSQL (nativeSql ); // put the id in the list into the HashSet <String> dbSet = new HashSet <String> (); for (String id: list) {dbSet. add (id) ;}// put the id in stuChooseCourseIds in dbSet for (String givenId: stuChooseCourseIds) {if (dbSet. add (givenId) {delIds. add (givenId) ;}return JsonUtils. toJson (delIds );}

HashSet is used for processing. First, once Hash is used, the search efficiency will certainly be greatly improved. Secondly, it cleverly utilizes the unique features of elements in the Set. In this way, the final execution efficiency has been improved by more than 20 times. The method that was originally unable to end in five minutes now only takes over a dozen seconds. This result is quite satisfactory.

 

When the data volume is small, the problem is not obvious. When the data volume is large, the problem arises. This problem also tells us that everything should be viewed from a developmental perspective. In this case, let's continue to look at the last version of the code.

 

Now the data volume is tens of thousands. If the time goes backwards and the data volume turns into one million, imagine putting millions of data in a list, without having to worry about the efficiency problem for the moment. Can you afford your memory? The answer is yes, and the memory will be burst. This problem must be solved. The final version is as follows:

<Pre name = "code" class = "java"> public String findDeletedStuChooseCourseIds (List <String> stuChooseCourseIds, String schoolCalendarId) {List <String> delIds = new ArrayList <String> (); // obtain idsString nativeSql = "select id from tableName where schoolcalendarid = '" + schoolCalendarId + "'"; List <String> list = commonEao.exe cute (nativeSql) of all stuchoosecourse ); // put the id in the list into the HashSet <String> dbSet = new HashSet <String> (); for (String id: list) {dbSet. add (id) ;}// put the id in stuChooseCourseIds in dbSet for (String givenId: stuChooseCourseIds) {if (dbSet. add (givenId) {delIds. add (givenId) ;}return JsonUtils. toJson (delIds );}

The method adds another parameter-the term of the academic year. The data generated by each semester remains unchanged, so that the execution efficiency of the method will not be affected no matter how long it takes.

 

Through this optimization, I really felt the charm of the data structure. Cool, have time to learn about the data structure!