Java memory overflow

Memory overflow and database lock table problems can be said to be a nightmare of developers. General Program Exceptions always tell when or when exceptions occur, according to the stack information, it is easy to locate a problem in the program. The memory overflow and lock table are not the case. The general phenomenon is that the system is getting slower and slower after the operation, until the system crashes, but it is not clear what operations occur, the time point of occurrence is also irregular. You cannot view logs or the database to locate the problematic code.

What's more serious is that memory overflow and database lock tables are not easily discovered in the system development and unit test stages. When the system is officially launched for a long time, the number of concurrent operations increases, as the data has accumulated, the system is prone to memory overflow or table locking. At this time, the system cannot stop or restart the system at will, which brings great difficulties for fixing bugs.

This article takes multiple projects developed and supported by the author as an example to share with you the Java memory overflow and database lock table detection and processing solutions encountered during the development process.

2. memory overflow Analysis
Memory overflow refers to the amount of memory that cannot be recycled or used in the application system. In the end, the memory used by the program is greater than the maximum memory that the virtual machine can provide. To solve the memory overflow problem in Java, we must first understand how Java manages the memory. Java memory management is the issue of object allocation and release. In Java, the memory allocation is completed by the program, and the memory release is completed by the Garbage Collection (GC, programmers do not need to call GC functions to release the memory, because different JVM implementers may use different algorithms to manage GC, and some may start working only when the memory usage reaches a certain level, there are also scheduled execution, and some are interrupted execution GC. However, GC can only recycle the space occupied by objects that are useless and no longer referenced by other objects. Java's memory garbage collection mechanism checks the reference chain from the main running objects of the program. After traversing it, it finds that the isolated objects that are not referenced are used as garbage collection.

There are many common causes of memory overflow:

L The amount of data loaded in the memory is too large, such as extracting too much data from the database at a time;

L The Collection class contains references to objects, which are not cleared after use, so that JVM cannot be recycled;

L there are endless loops or loops in the code that generate too many repeated object entities;

L bugs in third-party software used;

L The memory value of the startup parameter is set too small;

3. memory overflow Solution
Although memory overflow is tricky, there are also corresponding solutions, which can be solved step by step from ease to difficulty.

The first step is to modify the JVM startup parameters and directly increase the memory. This seems simple, but it is easy to ignore. By default, the JVM can use 64 MB of memory, and Tomcat can use 128 MB of memory by default, which is insufficient for systems with a little complexity. In a project, the "OutOfMemory" error is often reported because the default value of the startup parameter is used. Therefore, do not add the-Xms and-Xmx parameters.

Step 2: Check the error log and check whether there are other exceptions or errors before the "OutOfMemory" error. In a project, two databases are used for connection. The database connection used for sending text messages is managed using the DBCP connection pool. If you do not send text messages, you may change the database connection user name to an error, the log contains logs with many database connection exceptions. After a while, the "OutOfMemory" error occurs. After analysis, this is caused by a BUG in the DBCP connection pool. After the database fails to be connected, the connection is not released, so that DBCP reports the "OutOfMemory" error. After correct database connection parameters are modified, no memory overflow error occurs.

Checking logs is very important for analyzing memory overflow. By carefully checking the logs and analyzing the operations performed before memory overflow, You can roughly locate the problematic modules.

Step 3: arrange experienced programmers to check and analyze the code to find out where memory overflow may occur. Troubleshoot the following issues:

L check whether there is an endless loop or recursive call in the code.

L check whether there are large loops that repeatedly generate new object entities.

L check whether all data is obtained once in the database query. Generally, if 100,000 records are retrieved to the memory at a time, memory overflow may occur. This problem is relatively hidden. Before going online, the database has less data and is not prone to problems. After going online, there is more data in the database, and a single query may cause memory overflow. Therefore, we recommend that you query the database by page.

L check whether List, MAP, and other collection objects are not cleared after use. List, MAP, and other collection objects always have references to objects, so that these objects cannot be recycled by GC.

Step 4: Use the memory check tool to dynamically view the memory usage. After a project is launched, a memory overflow error occurs two days after the system is started. This situation is generally caused by slow memory leakage in the code, which cannot be solved by the above three steps. Therefore, you need to use the memory check tool.

There are many memory viewing tools, including Optimizeit Profiler, JProbe Profiler, JinSight, and Jconsole of Java1.5. Their basic working principles are similar. They monitor the application, release, and other actions of all objects when the Java program is running, and make statistics, analysis, and visualization of all information in memory management. Based on this information, developers can determine whether the program has a memory leakage problem. In general, the memory usage of a normal system after it is started is basically stable, instead of increasing without limit. Observe the size of the memory used during system operation. We can see that in the memory usage monitoring window, the basic rule is the sawing line. If the memory size continues to grow, it indicates that the system has memory leakage. Take a memory snapshot at intervals and compare and analyze the usage and reference information of objects in the memory snapshot to find out which class of objects are leaking.

By analyzing and processing the above four steps, we can basically handle the problem of memory overflow. Of course, this process also requires considerable experience and sensitivity, which must be accumulated in the actual development and debugging process.

In general, memory overflow is caused by poor code writing. Therefore, improving the code quality is the most fundamental solution. Some people think that it is wrong to first implement the function and correct it in the test phase when there is a BUG. If the quality of a product is determined in the manufacturing process, rather than in the quality inspection, the quality of the software has been determined in the design and coding stage, testing is only a verification of software quality, because it is impossible to identify all the bugs in the software.

 

Slave --------------------------------------------------------------------------------------------------------------------------------

 

There are many reasons, such:

1. Too large data volume; endless loops; too many static variables and static methods; recursion; unable to determine whether the referenced object is being referenced;

2. the VM does not recycle memory (Memory leakage );

To put it bluntly, a memory overflow occurs when the memory used by the program is larger than the maximum memory provided by the virtual machine. The memory overflow problem depends on the business and system size. For some systems, memory overflow may be uncommon, but some systems are still very common solutions,

One is to optimize the program code. if the business is huge and the logic is complex, we should minimize the reference of global variables. Releasing the reference when the program uses the variable will allow the Garbage Collector to recycle and release resources.
The second is physical solution, increase the physical memory, and then modify it through:-Xms256m-Xmx256m-XX: MaxNewSize = 256 m-XX: MaxPermSize = 256m

I. memory overflow type 
1. java. lang. OutOfMemoryError: PermGen space

JVM manages two types of memory, heap and non-heap. Heap is used by developers. It is created at JVM startup. Non-heap is reserved for JVM and used to store class information. Unlike the heap, GC does not release space during runtime. If the web app uses a large number of third-party jar files or the application has too many class files, but the MaxPermSize setting is small, exceeding this will also cause excessive memory usage and overflow, or during hot deployment of tomcat, the Environment loaded above will not be cleaned up, but the context will be changed to a new deployment, so there will be more and more non-heap content.

2. java. lang. OutOfMemoryError: Java heap space

The first case is a supplement. The main problem is that it appears in this case. The default space (-Xms) is 1/64 of the physical memory, and the maximum space (-Xmx) is 1/4 of the physical memory. If the remaining memory is less than 40%, the JVM will increase the heap to the Xmx setting value. If the remaining memory exceeds 70%, the JVM will reduce the heap to the Xms setting value. Therefore, the Xmx and Xms settings of the server should be set to the same value to avoid adjusting the size of the VM heap after each GC operation. If the physical memory is infinitely large, the maximum JVM memory is related to the operating system. Generally, 32-bit machines are between 1.5 GB and 3 GB, and 64-bit machines are not limited.

Note: If the Xms value exceeds the Xmx value, or the sum of the heap maximum value and the non-heap maximum value exceeds the physical memory or the maximum operating system limit, the server cannot be started.

Garbage collection GC role

JVM calls GC frequently. Garbage collection is mainly performed in two cases:

When the application thread is idle, and when the java memory heap is insufficient, GC is continuously called. If continuous collection fails to solve the problem of insufficient memory heap, an out of memory error is reported. This exception is determined by the system running environment, so it cannot be expected when it will appear.

According to the GC mechanism, program running may cause changes in the system runtime environment and increase the chance of GC triggering.

To avoid these problems, the program design and writing should avoid the memory usage and GC overhead of spam objects. It is shown that calling System. GC () can only be recommended that JVM recycle junk objects in memory, but not immediately,

One is that it cannot solve the problem of insufficient memory resources and increase GC consumption.

Ii. JVM memory zone Composition 
Java stack and stack

Java divides memory into two types: stack memory and heap memory.

1. The basic type variables defined in the function and the referenced variables of the object are allocated in the function stack memory;

2. Heap memory is used to store objects and arrays created by new.

When a variable is defined in a function (code block), java allocates memory space for the variable in the stack. When the scope of the variable is exceeded, java Automatically releases the memory space allocated for the variable. The memory allocated in the heap is managed by the java Virtual Machine's automatic garbage collector.

The advantage of heap is that the memory size can be dynamically allocated, and the lifetime does not need to be told to the compiler in advance because it dynamically allocates memory at runtime. The disadvantage is that the memory needs to be dynamically allocated at runtime, And the access speed is slow;

The advantage of stack is that the access speed is faster than the heap speed, but the disadvantage is that the data size and lifetime in the stack must be determined inflexible.

Java heap is divided into three areas: New, Old, and Permanent.

GC has two threads:

The newly created object is allocated to the New area. When the area is filled, it will be moved to the Old area by the GC auxiliary thread, when the Old area is full, the main GC thread is triggered to traverse all objects in the heap memory. The size of the Old area is equal to Xmx minus-Xmn.

Java stack Storage

Stack adjustment: the parameter is + usedefastackstacksize-Xss256K, indicating that each thread can apply for a stack space of kb.

Each thread has its own Stack

Iii. How to Set virtual memory in JVM
Tip: if 98% is used for GC and the available Heap size is less than 2% in JVM, this exception is thrown.

Tip: the Heap Size should not exceed 80% of the available physical memory. Generally, you must set the-Xms and-Xmx options to the same, and-Xmn to the-Xmx value of 1/4.

Tip: the initial memory allocated by JVM is specified by-Xms. The default value is 1/64 of the physical memory. The maximum memory allocated by JVM is specified by-Xmx. The default value is 1/4 of the physical memory.

By default, when the free heap memory is less than 40%, the JVM will increase the heap until the maximum limit of-Xmx. When the free heap memory is greater than 70%, the JVM will reduce the minimum limit of heap until-Xms. Therefore, the server generally sets-Xms and-Xmx to be equal to each other to avoid adjusting the heap size after each GC.

Tip: if the physical memory is infinitely large, the maximum JVM memory has a great relationship with the operating system.

Simply put, although the 32-bit processor has a controllable memory space of 4 GB, the specific operating system will impose a limit,

This limit is generally 2 GB-3 GB (1.5 GB-2 GB in Windows and 2 GB-3 GB in Linux ), the 64-bit and above processors will not be limited.

Tip: Note: If the Xms value exceeds the Xmx value, or the sum of the maximum heap value and the maximum non-heap value exceeds the limit of the physical memory or operating system, the server cannot be started.

Tip: Set NewSize and MaxNewSize to be equal. The size of "new" should not be greater than half of "old" because if the old area is not large enough to frequently trigger the "Main" GC, greatly reduced performance

JVM uses-XX: PermSize to set the non-heap memory initial value. The default value is 1/64 of the physical memory;

Set the maximum non-heap memory size by XX: MaxPermSize. The default value is 1/4 of the physical memory.

Solution: manually set Heap size

Modify TOMCAT_HOME/bin/catalina. bat

Add the following lines to "echo" Using CATALINA_BASE: $ CATALINA_BASE:

1. JAVA_OPTS = "-server-Xms800m-Xmx800m-XX: MaxNewSize = 256m"

Iv. Use of performance check tools 
Locate Memory leakage:

JProfiler is mainly used to check and track the performance of the system (limited to Java Development. JProfiler can monitor the system's memory usage and garbage collection and thread running status at any time to monitor the JVM running status and performance.

1. The memory of the application server is used unreasonably for a long time. The memory is often occupied at a high level, which is hard to be recycled to a low level;

2. The application server is extremely unstable and restarts every two days, sometimes even every day;

3. the Application Server often performs Full GC (Garbage Collection), which takes about 30-40 seconds. When performing Full GC, the application server does not respond to the customer's transaction request, it has a significant impact on system performance.

Because the development environment and product environment are different, this problem sometimes occurs in the product environment. Generally, you can use tools to track the memory usage of the system, in some cases, a large amount of memory may be used at some time, resulting in out of memory. In this case, we should continue to track whether there will be any decrease in the future,

If it remains high, it is certainly because the program causes memory leakage.

5. Features and solutions for unrobust code 
1. Release reference of useless objects as soon as possible. A good way is to enable the reference variable to be automatically set to null after exiting the active domain when using a temporary variable, suggesting that the Garbage Collector collects the object to prevent memory leakage.

For instances with pointers, jvm will not recycle this resource, because garbage collection will take null objects as garbage, improving the efficiency of GC collection mechanism;

2. In our program, a large amount of String processing is inevitable. To avoid the use of String, a large number of StringBuffer should be used. Each String object must occupy an independent area of memory;

1. String str = "aaa ";
3. String str2 = "bbb ";
5. String str3 = str + str2; // If str and str2 are not called after this execution, it will be stored in the memory and waited for Java gc to be recycled, if this happens too many times in the program, the above error will be reported. We recommend that you use StringBuffer when using strings instead of using strings, which saves a lot of overhead;

3. Use as few static variables as possible, because static variables are global and GC will not be recycled;

4. Avoid creating objects in a centralized manner, especially large objects. JVM will suddenly need a large amount of memory, which will inevitably trigger GC to optimize the system memory environment. The declared array space is displayed, and the number of applications is large.

This is a case for your warning:

JspsmartUpload is used for file upload, and java is often used during running. outofMemoryError: use the top command to check the process usage and find that the memory is less than 2 MB. It took a long time to find that it was a jspsmartupload problem. Decompile the source code file (class file) of the jspsmartupload component into a Java file:

1. M_totalBytes = m_request.getContentLength ();
2. M_binArray = new byte [m_totalBytes];

The variable m_totalBytes indicates the total length of the File Uploaded by the user, which is a large number. If you declare a byte array with such a large number and allocate memory space to each element of the array, and the m_binArray array cannot be released immediately, the JVM garbage collection is indeed faulty, the result is a memory overflow.

The reason why jspsmartUpload is to do this is that according to the http upload standard of RFC1867, a file stream is obtained without knowing the length of the file stream. The designer only knows the length of the file once after the servletinputstream operation, because no stream knows the size. Only when the length of the file is known can the user limit the length of the file to be uploaded. To save this trouble, the jspsmartUpload designer directly opens the file in the memory and determines whether the length complies with the standard, and writes the file to the server's hard disk. This results in memory overflow, which is just my guess.

When programming, do not apply for large space in the memory, because the memory of the web server is limited, and use stream operations as much as possible, such

1. Byte [] mFileBody = new byte [1, 512];
2. Blob vField = rs. getBlob ("FileBody ");
3. InputStream instream = vField. getBinaryStream ();
4. FileOutputStream fos = new FileOutputStream (saveFilePath + CFILENAME );
5. Int B;
6. While (B = instream. read (mFileBody ))! =-1 ){
7. Fos. write (mFileBody, 0, B );
8. }
9. Fos. close ();
10. Instream. close ();

5. Try to use the object pool technology to improve system performance. objects with a long life cycle have objects with a short life cycle, which may cause memory leakage, for example, when a large collection object has a large data volume of business objects, you can consider processing them in multiple parts and then solve the policy of releasing one piece.

6. Do not create objects in frequently called methods, especially avoid creating objects in loops. You can use hashtable as appropriate. The vector creates a group of object containers and then retrieves those objects from the containers without dropping them after each new operation.

7. It usually occurs when a large file is opened or too much data is taken with the database at a time, resulting in an Out Of Memory Error. In this case, the maximum data volume is calculated, set the minimum and maximum memory space values.