How to Reduce. NET application memory footprint

The most recent week is busy, the main work is to do a call "Keyboard Wizard" things, simply put a lot of data into memory, the data for quick retrieval, and then find the input conditions to match the best 10 records and show. Specific and the following two stocks of software related functions similar to:

Data in the form of text exists in the file, and the volume of data is large, there are nearly 200,000, each record has several fields, separated by delimiters. Using 60,000 records of the test data, the text file will be nearly 10M, the module loaded into memory and set up the cache, will probably occupy nearly 70-80m memory. After self takeover, the main task is to reduce memory consumption and improve the matching efficiency.

First, avoid creating unnecessary objects

Get the code, the first step is to look at the design document, and then breakpoint step-by-Step look at the code, probably understand the logic, found that there are some problems. The previous code processing process is probably the following:

Read the file to memory, instantiate

Retrieves a file based on criteria and stores it in result set 1

Match the results in result set 1, and store them in result sets 2

Sort the match by the result set 2, fetch the most matching 10 records, and then return

The process is in the same. But there are a lot of problems, the biggest problem is that the temporary variables store too many intermediate processing results, and these objects are immediately discarded after a query is completed, a large number of temporary objects bring a large amount of GC pressure. For example, when the user enters 1 in the input box, assuming the use of contains to match, then from 60,000 records to find 1 records may have more than 40,000, and then need to store more than 40,000 records in a temporary variable processing, to further calculate the matching of 40,000 records, It is then stored in a similar keyvaluepair set, the key is matched, then the set is sorted by key, then the first 10 best records are taken. As you can see, a large number of temporary variables are created in the middle, resulting in a surge in memory, which is reclaimed immediately after a large number of temporary objects are created, and the GC is stressed.

In a design document, only the 10 most matched records are required to return, and the previous solution does not seem to notice this. So after taking over, the first step is to streamline the process. The following are streamlined:

Read the file to memory, instantiate

Retrieves a file based on a condition, if it exists:

Calculates the matching degree.

To match the key, stored in a sortlist of only 11 capacity.

If the Sortlist collection adds more than 10 after the record is added, the last element is removed and the first 10 minimum (best match) record is always maintained.

After the traversal completes, returns the collection object

After this modification, reduce the amount of temporary data used to memory, the whole process, I just use a capacity of 11 sortlist structure to store the middle of the process, each time inserting an element, Sortlist help us arrange the order, and then remove the most mismatched one, that is, the last element ( From small to large sort, the more matches, the smaller the value. This consumption is mainly sortlist insertion, internal sorting and removal of records. When it comes to the choice of sortlist or sortdictionary, and then find some information, sortdictionary in the internal use of red-black tree implementation, sortlist using an ordered array implementation, in the internal sort are O (logn) under the premise , the time complexity of Sortdictionary O (LOGN) inserts and deletes elements is better than sortlist, but sortdictionary consumes more memory than sortlist. Basically, this is a balance between query speed and memory allocation, since there are only 11 objects to store, so there is little difference between the two. In fact, even if there is no such structure, they can be achieved, nothing more than a collection, each time add a, row a good order, and then remove the largest one. NET is easy to use because there are a lot of these powerful built-in data structures.

After this little modification, the memory footprint was reduced 1 time times, from the original 70-80m, reduced to 30-40m, in fact, this is to reduce the memory cost of one of the most basic principle, that is to avoid creating unnecessary objects.