Solution of memory overflow in Java Big Data processing (i)

Http://www.cnblogs.com/a757956132/p/4644910.html

http://my.oschina.net/songhongxu/blog/209951

One, memory overflow type

1. Java.lang.OutOfMemoryError:PermGen Space

The JVM manages two types of memory, heap, and non-heap. The heap is intended for developers, and is created at the start of the JVM, and the non-heap is left to the JVM for its own use to store the information for the class. Unlike heaps, the GC does not free up space during the run time. If the Web app uses a large number of third-party jars, or if the application has too many class files, and the MaxPermSize setting is small, exceeding this will cause the memory to overflow too much, or the Tomcat hot deployment will not clean up the previously loaded environment, Only the context is changed to the newly deployed, and the non-stockpiled content becomes more and more.

PermGen space is the full name of permanent Generation space, refers to the memory of the permanent storage area, this block of memory is mainly stored by the JVM class and meta information, class is loader will be placed in PermGen Space, unlike the heap area where the class instance (Instance) is stored, the GC (garbage Collection) does not clean up permgen space during the main program run time, so if you have a class in your application, PermGen space errors are most likely to occur when the Web server pre-compile the JSP. If you have a large number of third-party jars under your web app that are larger than the JVM's default size (4M), this error message will be generated.
One of the best configuration examples: (after I have verified, since the use of this configuration, there is no more tomcat dead situation)

Set java_opts=-xms800m-xmx800m-xx:permsize=128m-xx:maxnewsize=256m-xx:maxpermsize=256m

2. Java.lang.OutOfMemoryError:Java Heap Space

The first case is a supplement, and the main problem is that it appears in this situation. Its default space (that is,-XMS) is 1/64 of physical memory, and the maximum space (-XMX) is 1/4 of physical memory. If the memory is less than 40%,JVM the value of the heap to the XMX setting is increased, the memory remaining more than 70%,JVM will reduce the heap to XMS setting value. So the server's xmx and XMS settings should generally be set identically to avoid resizing the virtual machine heap after each GC. Assuming that the physical memory is infinite, then the maximum size of the JVM memory is related to the operating system, the general 32-bit machine is between 1.5g and 3g, and 64-bit will not be limited.

Note: If XMS exceeds the XMX value, or if the sum of the heap maximum and the non-heap maximum exceeds the physical memory or the maximum operating system limit, the server will not start up.

Role of garbage collection GC

The frequency with which the JVM calls the GC is still high, with garbage collection in two main cases:

When the application thread is idle, and the other is that the Java memory heap is not sufficient, the GC is constantly called, and if continuous recycling does not solve the problem of insufficient memory heap, it will report an out-of-storage error. Because this exception is determined by the operating environment of the system, it cannot be expected when it appears.

According to the GC mechanism, the operation of the program can cause changes in the operating environment of the system and increase the chance of the GC triggering.

To avoid these problems, the design and writing of the program should avoid the garbage object's memory footprint and the GC overhead. The display call System.GC () can only suggest that the JVM needs to recycle garbage objects in memory, but it does not have to be recycled immediately.

One is not able to solve the memory resource depletion situation, but also increase the consumption of GC.

Second, the JVM memory area composition

Simple to say heap and Stack in Java

Java has two kinds of memory: one is stack memory, the other is heap memory

1. The basic type variables and the reference variables of the objects defined in the function are allocated in the stack memory of the function;

2. Heap memory is used to hold objects and arrays created by new

When a variable is defined in a function (code block), Java allocates memory space for the variable in the stack, and when the scope of the variable is exceeded, Java automatically frees the allocated memory space for that variable, and the memory allocated in the heap is managed by the Java Virtual machine's automatic garbage collector

The advantage of a heap is that it can dynamically allocate memory size, and the lifetime does not have to tell the compiler beforehand because it allocates memory dynamically at run time. The disadvantage is to dynamically allocate memory at run time, the access speed is relatively slow;

The advantage of the stack is that the access speed is faster than the heap, and the disadvantage is that the data size in the stack and the lifetime must be deterministic without flexibility.

The Java heap is divided into three zones: New, old, and permanent

The GC has two threads:

The newly created object is assigned to the new zone, and when the area is filled, it is moved to the old area by the GC worker thread, and when the old area is filled, all objects in the heap memory that trigger the GC main thread traverse. The size of the old area equals Xmx minus-xmn

Java Stack Storage

Stack adjustment: Parameters have +usedefaultstacksize-xss256k, indicating that each thread can request 256k of stack space

Each thread has his own stack.

Iii. how the JVM sets up virtual memory

Tip: This exception message is thrown in the JVM if 98% of the time is used for GC and the available heap size is less than 2%.

Tip: The Heap Size does not exceed 80% of the available physical memory, generally the-XMS and-XMX options are set to the same, and the-XMX value of-xmn is 1/4.

Tip: The initial allocation of memory by the JVM is specified by-XMS, which defaults to the maximum allocated memory for 1/64;JVM of physical memory specified by-XMX, which defaults to 1/4 of the physical memory.

When the default free heap memory is less than 40%, the JVM increases the heap until the maximum limit of-xmx, and when the free heap memory is greater than 70%, the JVM reduces the heap until the minimum limit of-XMS. So the server generally sets-xms,-xmx equal to avoid resizing the heap after each GC.

Tip: Assuming that the physical memory is infinitely large, the maximum value of the JVM memory is very much related to the operating system.

In short, the 32-bit processor, although the controllable memory space has 4GB, but the specific operating system will give a limit,

This limit is generally 2GB-3GB (generally under Windows System is 2g-3g under the 1.5g-2g,linux system), and processors over 64bit are not limited

Tip: Note: If XMS exceeds the XMX value, either the sum of the heap maximum and the non-heap maximum exceeds the physical memory or the maximum operating system limit will cause the server to start up.

Tip: Set newsize, Maxnewsize equal, "new" should not be more than half the size of "old", because the old area if not enough to trigger "main" GC frequently, greatly reducing the performance

The JVM uses-xx:permsize to set the non-heap memory initial value, which defaults to 1/64 of the physical memory;

The maximum amount of non-heap memory is set by Xx:maxpermsize, which defaults to 1/4 of physical memory.

Workaround: Manually set the heap size

Modify Tomcat_home/bin/catalina.bat

Add the following line to the "echo" Using catalina_base: $CATALINA _base "":

Java_opts= "-server-xms800m-xmx800m-xx:maxnewsize=256m"

Iv. use of performance check tools

To locate a memory leak:

The Jprofiler tool is primarily used to examine and track the performance of the system (limited to Java development). Jprofiler can monitor the operation of the JVM and its performance by monitoring the memory usage of the system and monitoring the garbage collection, thread health and so on at all times.

1. Application server memory Long-term unreasonable occupation, memory is often in high occupancy, it is difficult to recover low;

2. The application server is extremely unstable, restarts almost every two days, and sometimes even restarts every day;

3. The application server often does full GC (garbage Collection), and the time is very long, about 30-40 seconds, the application server when doing full GC is not responding to the client's transaction requests, very affect system performance.

Because the development environment and the product environment will be different, causing the problem to occur sometimes in the production environment, you can usually use the tool to track the memory usage of the system, in some cases, perhaps at some point in time is indeed a large amount of memory caused out of memories, should continue to follow the next to see if there will be a decline ，

If it stays high, it's definitely a memory leak because of the program's cause.

Five, the characteristics of the non-robust code and solutions

1. Release the reference of the useless object as soon as possible. A good idea is to use a temporary variable when the reference variable is automatically set to null after exiting the active domain, implying that the garbage collector collects the object to prevent a memory leak.

For instances where pointers are still pointing, the JVM does not reclaim the resource because garbage collection will use null-valued objects as garbage, increasing the efficiency of the GC recovery mechanism;

2, our program is inevitably a lot of string processing, avoid the use of string, should use a large number of StringBuffer, each string object has to occupy an area of memory;

String str = "AAA";

String str2 = "BBB";

String STR3 = str + str2;//If you do this after STR, STR2 is not called later, it will be placed in memory waiting for the Java GC to be recycled, too much in the program such a situation will report the above error, It is recommended that you use StringBuffer when using strings, so you can save a lot of overhead by using string.

3, minimize the use of static variables, because the static variables are global, the GC will not be recycled;

4, avoid the centralized creation of objects, especially large objects, the JVM will suddenly need a lot of memory, it will inevitably trigger the GC to optimize the system memory environment, display the Declaration array space, and the number of applications is also very large.

This is a case for everyone to alert.

Using Jspsmartupload as a file upload, java.outofmemoryerror errors often occur during the operation,

Problem found after check: code in component

M_totalbytes = M_request.getcontentlength ();

M_binarray = new Byte[m_totalbytes];

The problem is that the totalbytes variable gets a large number, which causes the array to allocate a lot of memory space, and the array cannot be released in time. The solution can only be changed in a more appropriate way, at least not by triggering a outofmemoryerror solution. Reference: http://bbs.xml.org.cn/blog/more.asp?name=hongrui&id=3747

5, try to use the object pool technology to improve the system performance; objects with long life cycles are prone to memory leaks when they have short life-cycle objects, such as when large collection objects have large data volumes of business objects, consider chunking and then resolve a piece of policy that frees a piece.

6. Do not create objects in frequently invoked methods, especially when creating objects in loops. You can use Hashtable,vector to create a set of object containers and then fetch those objects from the container without discarding them each time you new

7, generally occurs in the opening of large files or with the database once took too much data, resulting in the state of the memory Error, it is likely to calculate the maximum amount of data, and set the required minimum and maximum memory space value.

Solution of memory overflow in Java Big Data processing (i)