Remember a Java memory overflow resolution process

Note: This article mainly records the process of solving the memory overflow problem rather than the specific problem.

Recently in writing a search engine, using inverted index structure for document retrieval, the basic idea of saving an index is to save the inverted list to an ordered map in memory (TREEMAP), and then write the inverted table in memory to disk when the memory occupies a certain threshold, and when the disk already has an index, The in-memory index is merged with the indexes on the disk to generate a new index, which is similar to a merge sort. The code for merging memory indexes and disk indexes is as follows:

     Public synchronized voidmerge () {Logutil.info ("Invertindex merge Start ..."); File F=NewFile (path);//This is the index file on the original disk
Disk exists index, merge disk index and memory indexif(F.exists ()) {String Outpath=path+ ". Temp"; File OutFile=NewFile (Outpath); TreeMap<string, treeset<long>> ramsnapshot=NULL; Ramsnapshot=RAM;//ram is a memory index, because RAM can be modified by other threads that add documents, so you save a snapshot and then empty the RAM, which is what you actually do with the snapshot Ram=NewTreemap<>(); BufferedReader Reader=NULL; PrintWriter writer=NULL; Try {
The merge process has 3 pointers: points to the current location of the memory index, where the original disk index is read, and where the new disk index is written Iterator<entry<string, treeset<long>>> ramiterator=Ramsnapshot.entryset (). iterator (); Reader=NewBufferedReader (NewFileReader (f)); Writer=NewPrintWriter (NewBufferedWriter (NewFileWriter (outFile)); Entry<string, treeset<long>> entry=ramiterator.hasnext ()? Ramiterator.next ():NULL; String Line=Reader.readline ();  while(entry!=NULL&&line!=NULL) {                    LongFreeram=runtime.getruntime (). Freememory ()/1000/1000; System.out.println ("Freeram:" +Freeram); String Ramword=Entry.getkey (); String Diskword=line.split (Separator1) [0]; String out=""; intC=Ramword.compareto (Diskword);
The merge process, because it is a two sequential list, uses a merge method similar to the merging sort, except that it needs to be merged into an inverted word if it encounters the same inverted word (merge both documents list)if(c==0) { TreeSet <Long> ramdocids= Entry.getvalue (); TreeSet<Long> diskdocids= This. Convertline2docids (line); TreeSet <Long> union= Ramdocids;                        Union.addall (diskdocids);                         out= This. Convertindex2line (Ramword, Union); Entry=ramiterator.hasnext ()? Ramiterator.next ():NULL; Line=Reader.readline (); }Else if(c<0) { out= This. Convertindex2line (Ramword, Entry.getvalue ()); Entry=ramiterator.hasnext ()? Ramiterator.next ():NULL; }Else{ out= This. Convertindex2line (Diskword, This. Convertline2docids (line)); Line=Reader.readline ();                } writer.println (out); } logutil.info ("Invertindex complex merge complete.");  while(Ramiterator.hasnext ()) {entry=Ramiterator.next (); String out= This. Convertindex2line (Entry.getkey (), Entry.getvalue ());                Writer.println (out); } logutil.info ("Invertindex Ram merge complete.");  while((Line=reader.readline ())! =NULL) {writer.println (line); } logutil.info ("Invertindex disk merge complete."); } Catch(Exception e) {Logutil.err ("Merge RAM index and disk index fail.", E); }finally {                Try {                    if(reader!=NULL) {reader.close (); }                    if(writer!=NULL) {writer.close (); }                } Catch(Exception E2) {logutil.err ("Release resource fail."), E2);}            } f.delete (); if(!outfile.renameto (NewFile (path)) {                Throw NewRuntimeException ("Rename temp file fail.")); }        }Else{//The index is not originally present on disk, write memory index directly to disk
Code slightly
    }

The main idea of the code is to maintain 3 pointers: The memory index is an ordered Treemap,iterator equivalent to a virtual pointer; The disk index is also ordered, Bufferdreader is equivalent to a virtual pointer The new index generated after the merge uses Bufferdwriter as a pointer. Then, using the idea of merge sorting, compare the size of the memory index words and the disk index words, which one will write to the new index and then forward its pointer to a bit, and if the two index words are equal, merge the document List of the two.

From the above description, the above code uses the memory should be O (1), because in memory, in addition to the memory index, the same time will only be read from the disk index line, but actually run, always in the merger times out GC overhead limit exceeded, This exception means that the JVM spent a lot of time (more than 98%) performing the GC but only freed up a very small amount of heap memory (less than 2%), in other words, a precursor to oom. According to my analysis of the memory consumption of the program, this situation is not normal.

Then add "-xx:+heapdumponoutofmemoryerror-xx:heapdumppath=c:/dump" parameter to the program, dump the memory information when the program has a memory overflow exception.

Use Eclipse mat to view the most memory-intensive objects:

The discovery is an instance of TreeMap, in this program I used TreeMap to save the memory index, and in the process of merging the index, the size of the memory index should be constant, then why overflow it?

Locate the problem in the red code by looking at the code:

This first points the Ramdocids to Entry.getvalue () and then the Union to Ramdocids, when the Union is actually pointing directly to Entry.getvalue (), and then the Union performs the AddAll operation: As we all know, the addall operation is to add elements to the object itself, where I originally intended to declare a local variable to save the list merge results and then into a new index file, but inadvertently but also modify the memory index resulting in a larger memory index.

OK, here the problem is solved, just need to

Treeset<long> Union=ramdocids;

Switch

Treeset<long> union=new treeset<> (ramdocids);

Can.

Review the process of resolving the memory overflow problem:

1. Analyze the complexity of the program space to see if the memory overflow is normal.
2. Add the JVM parameter to dump the memory snapshot when the program is in-store overflow.
3. Use eclipse mat to analyze the objects that occupy the most memory.
4. Find the appropriate object lookup problem in the source code.

Remember a Java memory overflow resolution process