The original problem is here-My Memory Leak diagnostic process

Since the company began to use Java as the main development language, the number of mixed applications of C ++ and Java has increased. The communication between Java and C ++ is mainly completed using JNI, which is no problem. For such a hybrid application project, the biggest nightmare is the memory leak diagnosis. Because the memory management mode of Java is very different from that of C ++, when debugging such a project, you must first distinguish between JavaCodeThe memory leak of is also the memory leak of C ++ code. For memory diagnosis, we need to first understand the meanings of some indicators and the use of tools in order to be rational.

Indicators:

Memory (Working Set): msdn description-the working set of a process is the set of pages in the virtual address space of the process that are currently resident in physical memory. the working set contains only pageable memory allocations; nonpageable memory allocations such as address wing extensions (AWE) or large page allocations are not supported in the working set.

A dedicated working set of a process refers to the Virtual Memory Page set where the current process is resident in the physical memory. It only contains memory areas that can be paged; memory areas that cannot be paged, such as awe (an applicationProgramYou can directly manipulate technologies larger than 4G physical memory) or LPA (mainly used for large physical memory of servers, which is generally useful for 64-bit Systems) will not be included in a dedicated work set.

Virtual Bytes: the virtual memory used by the current process. This indicator contains all the memory page files, such as the page files in the disk swap area and the loaded library files.

Private Bytes: the size of the private memory allocated by the current process, excluding the memory shared to other processes.

In general, if your application requires a small amount of memory and is active, the memory leakage is obvious, you can monitor the working set to see whether there is a memory leakage. However, if the application is complex, there are many modules, and the changes in different time periods are relatively large, the problem cannot be identified simply based on the working set, because some page files will be swapped in the disk buffer at a certain time. Therefore, we need to analyze the virtual bytes and private bytes indicators. However, this is not absolutely accurate, because sometimes, for example, there are many memory fragments, and applications often request large contiguous memory blocks, which may also increase virtual bytes. Therefore, in the actual environment, we also need to know the memory usage characteristics of the application to determine whether there is a memory leak problem.

After the indicators are introduced, the following tools are introduced:

For Java programs, the better tool for monitoring memory is jvirsualvm. This tool comes with Java. It can monitor local or remote Java applications, or programs such as system services. You can find it in the bin directory of JDK. Others include jconsole and eclipse mat.

For C ++ programs, there are more tools. Here I mainly use the IIS debug diagnostics tool, which was originally used for the diagnosis of IIS applications and is very convenient for monitoring system services. It can also perform Automatic Memory Leak analysis and generate reports. It is helpful for the diagnosis of memory leak. Of course, I also used two applications, vmmap and rammap. These two programs were originally developed by the author of system internal and have now been taken down by Microsoft. One of the two files is used to view the virtual memory allocation of the process, and the other has to view the physical memory. The last tool is process hacker, which helps us to learn more about the process memory allocation, handle allocation, module loading, and thread count. Of course, processexplorer can also do the same job, but if you want to view the content of the memory block, you also need the combination of windbg.

Basically everything we need is ready, so the next step is the real debugging journey. Here I will first introduce the program I want to debug as an enterprise-level backup service based on Tomcat, and the job of Java is to perform statistics and management based on the C ++ module. For such an application, memory leak is a headache because it cannot be solved through debugging. If memory leak appears, we need to distinguish between Java code and C ++ code. Because Java is a language with a garbage collection mechanism, memory leak is not easy to check. So what is Memory Leak for Java? An object cannot be recycled, that is, a leak. For example, if the object is put in a singleton list, as long as the Singleton is not released, the object will always exist in the memory. For the Tomcat program, we need to configure several parameters for jvisualvm monitoring:

-DCOM. Sun. Management. JMX remote. Port = 8086
-DCOM. Sun. Management. jmxremote. SSL = false
-DCOM. Sun. Management. jmxremote. Authenticate = false

In this way, a JMX connection can be established in jvisualvm for Tomcat. Then, we need to perform heap dump at different time points, then, check whether memory leak occurs by comparing the object changes in the two periods. You need to know more about object allocation in your application. In Java, heap and permgen are the main considerations for memory. heap is used for new objects, and permgen is used to store class and meta data content.

After unremitting efforts, we finally found that no timeout was set for HTTP connection in our program, which led to a lot of threads accessing the Web service in persistent connection mode. After completing the memory leak issue in Java, the entire c ++ module should be resolved. After a period of observation, it is found that the size of the working set will decrease at a certain time point, but the virtual bytes and private bytes increase in stages. This indicates that a block of code may be generated when a large block of memory is requested, but this does not prove to be a leak. This code is not necessarily in C ++. To find out whether the problem is in C ++ or Java, I need to track the memory usage of Java (the memory check function of process hacker is used here ).

In the end, I found that the virtual memory of the Java part has not changed and remains within the maximum memory size we set. In this way, we can start working on the C ++ side. First use vmmap to check the memory usage and find that the memory read and write volume is very large in one place, which proves that some programs are very intensive to access a memory. The next step is to use the debug diagnostics tool to track Tomcat's service. first create a memory leak rule and then start tracking. After a period of time, when the memory changes significantly, select memory dump for automatic analysis. You will get an approximate memory Analysis Report and the report will provide a possible Memory Leak module. Based on this information, you can choose to check the code of a module to determine whether the leak is found.

In the end, the problem was solved by the encryption and decryption function. In a function, malloc uses a piece of memory to store strings and then encrypts the strings, copy the data to the new encrypted memory and return the data directly. Instead, go to free after the returned code. In this way, each successful encryption will leak a piece of memory, because the encryption function is called only at a specific time, so the memory increases in stages.

By now, the entire memory leak diagnosis process is over. My feeling is that there must be both technology and luck to solve this problem! (The figure above is found and does not describe the actual situation, so you can see it)

All tools involved can be found through Google.