Java program's continuous Full GC processing experience

1. Problems Found

Recently, when I checked the crawler program running, I found that the collection speed was very slow. When I checked the cpu usage, I found that one kernel was always 100%. It is reasonable to say that this should not happen to a multi-threaded program. The first reaction was to use jstack-l [pid] to dump the thread to view the stack status, but no exception was found. Then we should use jmap-heap [pid] to print out the memory usage. The strange thing is that the connection may fail several times. After printing, we can see that the old generation is full. By increasing the memory size, we can see that this is the case in a short time. It is really strange.

2. in-depth investigation

It seems that the problem is tricky, so we have to analyze it in a more detailed way. On the one hand, GC logs are enabled, and jstat-gcutil [pid] 1000 is used to track GC at runtime. It is strange that the space of the old generation is growing slowly, but the memory usage of the new generation is 100%, and the old generation is also 100%, and then it is in full gc, but the memory is not reduced at all, the entire process lasted for 5 minutes. I forgot to save it. Tragedy ...)

It seems that the object has exploded, but the program has been running for a long time. If there is such a serious bug, it should have been discovered for a long time. For the sake of rigor, I used jmap-histo [pid] to print out the program object, and the results were very surprised.

 

  
num     #instances         #bytes  class name 
  
--------------------------------------------- 
  
  1:      30110820     1204432800  org.jsoup.parser.ParseError 
  
  2:         33076      156025088  [Ljava.lang.Object; 
  
  3:         68836       98796360  [C 
 

There are so many ParserError objects in the system. This is a Jsoup open source package used in the program. The name looks like an Exception, but only when you look at the source code can you find that it is actually a Class. Where did this thing pop up ?! I printed the thread stack information when the phenomenon occurred, and found the location where the object was generated.

Find the corresponding implementation in the source code, mainly by calling this method:

 

  
private boolean trackErrors = true;
  
......
  
void error(TokeniserState state) { 
  
    if (trackErrors) 
  
        errors.add(new ParseError("Unexpected character in input", reader.current(), state, reader.pos())); 
  
} 
 

Here there is a trackErrors switch, the default value is true, but after looking for it, there is no outermost method to control it, and the errors object is also a private value without any calls, it may be tested by the author himself.

3. troubleshooting

The cause investigation is clear, and the handling is simple. Directly change the default value of trackErrors in the source code to false, confirm the other calling methods, re-compile and package, and replace the original package.

4. Summary

Find the problem and analyze the possible causes first, and then use tools to locate the problem. In many cases, data is more reliable than experience.