The original address: http://blog.csdn.net/xianmiao2009/article/details/49254391 memory overflow and Database lock table problem, can be said to be the nightmare of developers, general program exceptions, It is always possible to know when or where an exception occurred, and it is easy to locate a problem somewhere in the program based on the stack information. Memory overflow and lock table is not, the general phenomenon is the operation of the general time after the system is more and more slow, until the crash, but it is not clear what operations appear, the occurrence of the point of time is not regular, view the log or view the database can not locate the problem code. More serious is the memory overflow and the database lock table in the system development and unit testing phase is not easy to find, when the system formally on the general time, the operation of the concurrency, the data also accumulated some, the system is prone to memory overflow or lock table phenomenon, and at this time the system can not be idle or restart, It's a lot of difficulty to fix bugs. In this paper, I develop and support a number of projects as an example, with you to share in the development process encountered in the Java memory overflow and database lock table detection and processing process. 2. Memory overflow analysis memory overflow refers to the application system in the presence of the memory can not be recycled or use too much memory, and ultimately make the program run to use more memory than the virtual function provides maximum memory. In order to solve the memory overflow problem in Java, we must first understand how Java manages memory. The memory management of Java is the allocation and release of objects. In Java, the allocation of memory is done by the program, and the release of memory is done by the garbage collector (garbage COLLECTION,GC), programmers do not need to call the GC function to free memory, because different JVM implementations may use different algorithms to manage the GC, Some of the memory usage to a certain extent, the GC began to work, there are timed execution, there are some interrupt execution GC. However, the GC can only reclaim space occupied by objects that are useless and are no longer referenced by other objects. The memory garbage collection mechanism of Java is to check the reference chain from the main running object of the program, and when it is traversed, it is found that there are no referenced orphaned objects as garbage collection. There are a number of reasons for memory overflow, which are common in the following ways: L The amount of data loaded in memory is too large, such as fetching too much data from the database at one time; l There are references to objects in the collection class that are not emptied after use, so that the JVM cannot be reclaimed; L There is an object entity in the code that has a dead loop or a loop that produces too many duplicates; L Use the firstBug;l startup parameters in the three-party software too small to set the memory value; 3. Memory overflow resolution memory overflow although very tricky, but there are corresponding solutions, you can follow from easy to difficult, step to solve. The first step is to modify the JVM startup parameters and increase the memory directly. This may seem simple, but it can easily be overlooked. The default memory that the JVM can use for 64m,tomcat is 128MB, which is not enough for a slightly more complex system. In a project, the "OutOfMemory" error is often reported because of the default value used by the startup parameter. Therefore, the-XMS,-XMX parameter must not forget to add. In the second step, check the error log to see if there are any other exceptions or errors before the "OutOfMemory" error. In a project, the use of two database connection, which is dedicated to send text message database connection using DBCP Connection pool management, the user does not send text messages, intentional database connection user name is wrong, so that the log has many database connection Exception log, after a period of time, there will be "OutOfMemory" Error. After analysis, this is due to the DBCP connection pool bug caused, the database connection is not on, the connection is not released, and eventually make the DBCP "OutOfMemory" error. After modifying the correct database connection parameters, no more memory overflow errors occur. Viewing logs is important for analyzing memory overflows by carefully reviewing the logs to analyze what has been done before the memory overflow, and can roughly locate the problematic module. The third step is to arrange for experienced programmers to walk through and analyze the code to find out where a memory overflow might occur. Focus on the following points: L Check for dead loops or recursive calls in your code. L Check if there are cycle duplicates of the new object entity. L Check to see if there is a query for all the data in the database query. In general, if you take 100,000 records to memory at a time, you can cause a memory overflow. This problem is more covert, before the online, the database less data, not prone to problems, on-line, the database more data, a query may cause memory overflow. Therefore, query the database query as much as possible by paging. L Check if the list, map, and other collection objects are not cleared after use. Collection objects such as List, map, and so on will always have a reference to the object so that they cannot be reclaimed by GC. Fourth step, use the memory viewing tool to dynamically view memory usage. Once a project is online, each system startsIn two days, a memory overflow error occurs. This is typically a slow memory leak in the code, which is not resolved with the three steps above, which requires the use of a memory viewing tool. There are many memory viewing tools, notably: Optimizeit Profiler, JProbe Profiler, Jinsight and Java1.5 jconsole, and more. They basically work the same principle, all of the Java program to monitor the operation of all objects application, release and other actions, the memory management of all the information to statistical, analysis, visualization. This information can be used by developers to determine whether a program has a memory leak problem. Generally speaking, a normal system after its start-up, its memory consumption is basically stable, and should not be unrestricted growth. Continuously observing the size of the memory used by the system when it is running, you can see a jagged graph of the basic rules in the Memory Usage monitoring window, and if the memory size continues to grow, the system has a memory leak issue. By taking a memory snapshot at a time interval, and then analyzing the usage and reference of objects in the memory snapshot, we can find out which class objects are leaking. Through the analysis and processing of the above four steps, the problem of memory overflow can be handled basically. Of course, in these processes also need considerable experience and sensitivity, need in the actual development and commissioning process accumulated. In general, the resulting memory overflow is due to poor code writing, so improving the quality of the code is the most fundamental solution. Some people think that the implementation of the function first, there is a bug in the testing phase of the correction, this idea is wrong. Just as the quality of a product is determined in the process of manufacturing, rather than in quality testing, the quality of the software has been decided during the design and coding stages, and testing is only a validation of the quality of the software, since it is impossible to find all the bugs in the software. --------------------------------------------------------------------------------------------------------- There are many reasons for----------------------- , such as: 1. The amount of data is too large; a dead loop; static variables and static methods are too many; recursive; Unable to determine whether the object is referenced; 2. Virtual machines do not reclaim memory (memory leaks); To put it bluntly, a memory overflow occurs when the program runs more memory than the virtual function provides. Memory overflow problem depends on the business and system size, for some systems may be memory overflow is not common, but some systems are very common solution, one is to optimize the program code, if the business is large, complex logic, minimize the global variable reference, let the program makeReleasing the reference when the variable is exhausted allows the garbage collector to reclaim and free up resources. The second is physical solution, increase physical memory, and then through:-xms256m-xmx256m-xx:maxnewsize=256m-xx:maxpermsize=256m modification One, memory overflow type 1, Java.lang.OutOfMemoryError:PermGen SPACEJVM manages two types of memory, heap and non-heap. The heap is intended for developers, and is created at the start of the JVM, and the non-heap is left to the JVM for its own use to store the information for the class. Unlike heaps, the GC does not free up space during the run time. If the Web app uses a large number of third-party jars, or if the application has too many class files, and the MaxPermSize setting is small, exceeding the amount of memory that can cause overflow, or if the Tomcat hot deployment does not clean up the previously loaded environment, only the context Change to a newly deployed, non-stockpiled content will be more and more. 2. Java.lang.OutOfMemoryError:Java heap Space The first situation is a supplement, and the main problem is that it appears in this situation. Its default space (that is,-XMS) is 1/64 of physical memory, and the maximum space (-XMX) is 1/4 of physical memory. If the memory is less than a maximum of $, the JVM will increase the heap-to-Xmx setting and the JVM will reduce the heap-to-Xms value by more than the memory remaining. So the server's Xmx and Xms settings should generally be set identically to avoid resizing the virtual machine heap after each GC. Assuming that the physical memory is infinite, then the maximum size of the JVM memory is related to the operating system, the general 32-bit machine is between 1.5g and 3g, and 64-bit will not be limited. Note: If Xms exceeds the XMX value, or if the sum of the heap maximum and the non-heap maximum exceeds the physical memory or the maximum operating system limit, the server will not start up. Garbage collection GC The role of the JVM calls GC is still very high, the main two cases of garbage collection: When the application thread is idle, the other is the Java memory heap is insufficient, will continue to call the GC, if the continuous recovery can not solve the problem of insufficient memory heap, will be reported out of a memory error. Because this exception is determined by the operating environment of the system, it cannot be expected when it appears. According to the GC mechanism, the operation of the program can cause changes in the operating environment of the system and increase the chance of the GC triggering. In order to avoid these problems, the design and writing of the program should avoid the garbage objectMemory consumption and GC overhead. The display call System.GC () can only suggest that the JVM needs to recycle the garbage object in memory, but it does not have to be recycled at once, one that does not solve the memory depletion situation, and also increases GC consumption. Ii. JVM Memory Area composition Simple Java heap and stack Java divide memory into two kinds: one is stack memory, the other is heap memory 1. The basic type variables and the reference variables of the objects defined in the function are allocated in the stack memory of the function; 2. Heap memory is used to hold objects created by new and arrays in a function (code block) to define a variable, Java in the stack to allocate memory space for this variable, when the scope of the variable is exceeded, Java will automatically release the allocated memory space for the variable, the memory allocated in the heap by the The advantage of a Java Virtual machine's automatic garbage collector to manage the heap is that it can dynamically allocate memory size, and the lifetime does not have to tell the compiler beforehand because it allocates memory dynamically at run time. The disadvantage is to dynamically allocate memory at run time, the access speed is slow, the advantage of the stack is faster than the heap, the disadvantage is that there is a stack of data size and lifetime must be determined without flexibility. The Java heap is divided into three zones: new, old, and PERMANENTGC have two threads: the newly created object is assigned to the area of new, and when the area is filled it is moved to the old area by the GC worker thread, and when the old area is filled, all objects in the heap memory that trigger the GC main thread traverse. The old area is equal to Xmx minus the-xmnjava stack: The parameter has a +usedefaultstacksize-xss256k, which means each thread can request 256k of stack space each thread has his own stack. Third, how the JVM sets up virtual memory Tip: This exception will be thrown in the JVM if 98% of the time is used for GC and the available HEAP size is less than 2%. Tip: The Heap Size does not exceed 80% of the available physical memory, generally the-XMS and-XMX options are set to the same, and the-XMX value of-xmn is 1/4. Tip: The initial memory allocated by the JVM is specified by-XMS, which defaults to the 1/64 of the physical memory, and the maximum allocated memory for the JVM is specified by-XMX, which defaults to 1/4 of the physical memory. When the default free heap memory is less than 40%, the JVM increases the heap until the maximum limit of-xmx, and when the free heap memory is greater than 70%, the JVM reduces the heap until the minimum limit of-XMS. So the server generally sets-xms,-xmx equal to avoid resizing the heap after each GC. Tip: Assuming that the physical memory is infinitely large, the JVM's memoryLarge values are very much related to the operating system. Simply put, 32-bit processor Although the controllable memory space has 4GB, but the specific operating system will give a limit, this limit is generally 2GB-3GB (generally speaking, Windows system is 1.5G-2G, Linux system is 2g-3g), and There is no limit to the number of processors over 64bit: note: If XMS exceeds the XMX value, or if the sum of the heap maximum and non-heap maximum exceeds the physical memory or the maximum limit of the operating system, the server will not start up. Hint: set NewSize, Maxnewsize equal, "new" size is best not more than half of "old", because the old area if not enough to trigger the "main" GC frequently, greatly reducing the performance JVM uses-xx:permsize set non-heap memory initial value, The default is 1/64 of physical memory, the maximum non-heap memory is set by Xx:maxpermsize, and the default is 1/4 of physical memory. Workaround: Manually set the Heap size to modify Tomcat_home/bin/catalina.bat on the "echo" Using catalina_base: $CATALINA _base "" Add the following line above: java_opts= "- server-xms800m-xmx800m-xx:maxnewsize=256m " Performance Check tool use Locate memory leak: The Jprofiler tool is primarily used for inspection and tracking systems (Java The performance of the developed). Jprofiler can monitor the operation of the JVM and its performance by monitoring the memory usage of the system and monitoring the garbage collection, thread health and so on at all times. 1. Application server memory for a long time unreasonable occupation, memory is often high occupancy, difficult to recover to low; 2. The application server is extremely unstable, restarts almost every two days, and sometimes even restarts every day; 3. Application servers often do full GC (garbage Collection), and the time is very long, about 30-40 seconds, the application server when doing the full GC is not responding to the client's transaction requests, very affect system performance. Because the development environment and the product environment will be different, causing the problem to occur sometimes in the production environment, you can usually use the tool to track the memory usage of the system, in some cases, perhaps at some point in time is indeed a large amount of memory caused out of memories, should continue to follow the next to see if there will be a decline , if it stays high, it must be due to proceduralA memory leak is caused. Five, the characteristics of the non-robust code and solutions 1, release the reference of the useless object as soon as possible. A good idea is to use a temporary variable when the reference variable is automatically set to null after exiting the active domain, implying that the garbage collector collects the object to prevent a memory leak. For instances where pointers are still pointing, the JVM does not reclaim the resource because garbage collection increases the efficiency of the GC recovery mechanism by using the null object as garbage; 2. We will inevitably use a lot of string processing in our program, avoid using string, and use StringBuffer extensively. Each string object has to occupy an area of memory independently; string str = "AAA"; string str2 = "BBB"; string STR3 = str + str2;//If after the execution of STR, STR2 is not called later, it will be placed in memory waiting for the Java GC to recycle, too much in the program to appear such a situation The situation will be reported above the error, it is recommended that you use a string when you can use StringBuffer do not use string, which can save a lot of overhead, 3, as little as possible with static variables, because the static variables are global, GC will not be recycled; 4. Avoid concentrating on creating objects especially This is a large object, the JVM will suddenly need a lot of memory, it will inevitably trigger the GC to optimize the system memory environment, display the Declaration array space, and the number of applications is also very large. This is a case for everyone to alert: Use Jspsmartupload for file upload, Now run the process often appear java.outofmemoryerror error, with the top command to see process usage, found that memory is less than 2M, took a long time, found to be jspsmartupload problem. Jspsmartupload components of the source file (class file) compiled into Java files, wake up: M_totalbytes = M_request.getcontentlength (); m_binarray = new Byte[m_totalbytes]; Variable m_totalbytes represents the total length of the file uploaded by the user, which is a large number. If you use such a large number to declare a byte array and allocate memory space for each element of the array, and M_bThe inarray array cannot be released immediately, and the JVM's garbage collection does have a problem, resulting in a memory overflow. Jspsmartupload Why to do this, there is his reason, according to the RFC1867 HTTP upload standard, get a file stream, do not know the length of the file stream. Designers if you want the length of the file, only the operation ServletInputStream once know, because no stream is known size. Only if you know the length of the file, you can limit the length of files uploaded by users. To save this trouble, the Jspsmartupload designer opens the file directly in memory, judging whether the length meets the standard and writes to the server's hard drive. This creates a memory overflow, which is just one of my guesses. So when programming, do not request large space in memory, because the Web server memory is limited, and as far as possible the use of flow operations, such as byte[] mfilebody = new byte[512]; blob vfield= rs.getblob ("Filebody"); InputStream instream=vfield.getbinarystream (); FileOutputStream fos=new FileOutputStream (savefilepath+cfilename); int B; while ((b =instream.read (Mfilebod Y)) =-1) { Fos.write (mFil EBODY,0,B); &NBSp } fos.close (); Instream.close (); 5, using the object pool technology to improve the performance of the system; objects with a long life span are prone to memory leaks when objects with short lifecycles, such as large collection objects with large data volumes of business objects, can consider chunking and then resolve a piece of policy to release a piece. 6. Do not create objects in frequently invoked methods, especially when creating objects in loops. It is possible to use Hashtable, vectors to create a set of object containers, and then to fetch those objects from the container, without having to discard 7 each time after new, usually in the case of opening large files or taking too much data with the database at one time, resulting in the state of the out of Memory Error, At this point, you probably want to calculate the maximum amount of data, and set the minimum and maximum memory space values.
[Go]java memory overflow explained and solution
