The Java.lang.OutOfMemoryError of Mina

The problem with the Mina framework was discovered during the test: when the Mina file was transferred more than a certain value (such as 55m) or the number of successive files was too frequent, the memory overflowed:

Org.apache.mina.filter.codec.ProtocolEncoderException:java.lang.OutOfMemoryError:Java Heap Space

Atorg.apache.mina.filter.codec.ProtocolCodecFilter.filterWrite (protocolcodecfilter.java:217)

Atorg.apache.mina.common.support.AbstractIoFilterChain.callPreviousFilterWrite (abstractiofilterchain.java:361)

atorg.apache.mina.common.support.abstractiofilterchain.access$1300 (abstractiofilterchain.java:53)

Atorg.apache.mina.common.support.abstractiofilterchain$entryimpl$1.filterwrite (AbstractIoFilterChain.java:659)

Atorg.apache.mina.common.support.abstractiofilterchain$tailfilter.filterwrite (AbstractIoFilterChain.java:587)

Atorg.apache.mina.common.support.AbstractIoFilterChain.callPreviousFilterWrite (abstractiofilterchain.java:361)

Atorg.apache.mina.common.support.AbstractIoFilterChain.fireFilterWrite (abstractiofilterchain.java:355)

Atorg.apache.mina.transport.socket.nio.SocketSessionImpl.write0 (socketsessionimpl.java:166)

Atorg.apache.mina.common.support.BaseIoSession.write (baseiosession.java:177)

Atorg.apache.mina.common.support.BaseIoSession.write (baseiosession.java:168)

Atcom.taobao.forest.server.DefaultPushTimeTask.pushcachetothesession (defaultpushtimetask.java:441)

1) Start is to try to use the conventional method to try to analyze Mina in memory overflow when what is accounted for so much memory can not be released, so in the JBoss boot parameters that add two parameters -xx:heapdumppath=\tmp-xx:+ Heapdumponoutofmemoryerror, the function is to dump the current memory image to/tmp when the outofmemoryerror occurs, and then dump the memory image file down to the local mat analysis, However, it is strange that the results of the analysis have not found a memory overflow problem.

2) After the Internet to check some information, only to find that Mina is not used heap memory (heap), but the use of native direct memory

the so-called local direct memory is not part of the data area when the virtual machine is running, it is native memory Rather than the area that the VM directly manages.

add New in JDK1.4 Into the nio class, which introduces a channel-to-buffer I/O method that can be directly assigned to the native native function library memory , and then through a Directbytebuffer object stored in the Java heap as this block . This can significantly improve performance in some scenarios because it avoids copying data back and forth between Java pairs and the native heap. Obviously this machine directly is not limited by the Java heap size, but it is memory that must still be subject to the native physical memory (including swap areas or Windows virtual memory ), the general server administrator configures JVM parameter, depending on the actual memory sets parameter information such as-XMX, but often ignores direct memory , making each Span style= "COLOR:WINDOWTEXT;" The sum of the memory area is greater than the physical memory limit (including physical and operating system-level limitations), which results in dynamic scaling Outofmemoryerror exception .

Also, according to the JVM specification, the maximum number of local direct memory is set in the following order:
(1) Specify a value by-xx:maxdirectmemorysize=<size>
(2) if (1) is not satisfied, then take maxmemory, that is, by the value set by-XMX;
(3) if (1), (2) are not satisfied, then take the default value: 64M;

According to the above knowledge, combined with this test situation, the problem is basically the bottom:

In our test routine machine, the system starts the time set-XMX 3072m, does not set the local direct memory maximum by the-xx:maxdirectmemorysize, therefore the local direct memory maximum value is the-XMX set value 3072m, the entire system physical memory is 4G, Remove the memory that the system process occupies, and the remaining physical memory plus swap space is close to 3G. Assuming that the JVM's heap size occupies 1.5g,direct memory using 1.5G, this time the program applies 1100 m of direct memory, in which case either the JVM heap size or the direct memories do not meet the conditions that trigger the GC. The JVM then requests the OS to allocate memory, but the OS has no memory to allocate, so it throws a outofmemoryerror error.

Therefore, when using the NIO framework, it is important to note that:
If the NIO framework uses a direct deposit, you need to carefully set the JVM operating parameters, preferably with-xx:maxdirectmemorysize settings, otherwise you have to know that you set the-XMX not only set the maximum size of the heap, it is also direct The maximum value of memory;

Let's add a little bit of nio and oom knowledge:

First, the understanding of the concept of available memory

On a 32-bit machine, the maximum amount of physical memory that can be addressed by the CPU is 4G, which is no longer visible beyond 4G. "The PAE support is ignored here, and if an AWE (Windows) or Mmap (Linux)-class scenario is used in the process, this is a temporary"

This 4G of physical memory space is divided into user space and kernel space. By default, Windows is divided by 50:50, and Linux defaults to 3G of user space and kernel space of 1G.

So the physical memory space available for a process, under the linux32 bit machine, is 3G. In the 64-bit machine, basically can be considered without any restrictions, the principle is very simple ...

Whether it's Linux or Windows, the available memory space consists of physical memory +swap/virtual memory. Called swap "swap space" on Linux, Windows is called virtual memory, essentially taking a piece of disk as physical memory. The program is not concerned with the use of physical memory, or swap, the program operates a virtual address space, the OS then maps the virtual address space to physical memory, files, or other. Whether it's manipulating physical memory or swap, it's completely transparent to the program.

swap/virtual memory when it will be used, I am not fully aware of this, but one thing should be correct, is the process of the new application of memory, not in the swap/virtual memory allocation, but directly in the physical memory allocation. When memory is tight, the OS swaps the memory occupied in the active process from the physical memory and onto the swap/virtual memory (and sometimes even the memory is not tense). When the process resumes activity, the OS reads the data from the swap/virtual memory space and puts it into physical memory

So when you need to analyze and calculate the memory space that the process needs to occupy, you can simply ignore the concept of swap/virtual memory "This needs to be further demonstrated!" 】

Second, the JVM management of memory

Drawing a picture, it is easy to understand that the following circle represents all the memory space occupied by the JVM process, divided into three parts:

1. Heap Space

Including younger, older generations, persistent domains "with the Sun HOTSPOT virtual machine implementation as an example, other virtual opportunities are different, such as IBM's virtual machine, so-called" persistent domain "is not in the heap allocation, but in local memory"

If this space is not enough, it will throw java.lang.OutOfMemoryError.

2. Stack space

Each thread will have a separate stack space, JDK5.0 previously default as if the 256k,jdk5.0 default is 1M, a large number, can be set by-XSS. If this space is not enough, it will throw java.lang.StackOverflowError.

3. Local memory

The memory that the JVM process can use, after removing the heap and stack space, the rest is the local memory

The above three spaces add up to memory, which is all the memory used by the final JVM process. If it is under a 32-bit machine, cannot exceed the user space size, that is 3G; under 64-bit machines, it depends on the size of the physical memory.

Another reminder that in the event of insufficient memory, blindly increase-xms and-xmx, it is likely to backfire, the rationale should be very obvious. Need to see the type of Oom, is the heap is insufficient, or the stack (StackOverflow) is insufficient, or local memory is not enough native memories. The JVM will generally have enough information to prompt.

Third, NIO's direct memory allocate

I understand that NiO's direct memory allocation "DMA" should be to allocate memory from the local memory area. As previously said, if you do not use the-xx:maxdirectmemorysize setting, then it will use the-XMS settings, for the daily test environment, for example, in this case, the DMA needs 3G, the heap also needs 3G, it is obvious that the two space can not be so large memory, So either the heap space is squeezed, the 3G is not available, or the DMA is not getting enough space

Looking at the bugs thrown out by the JVM should be caused by the squeezing of the heap space. If the local memory is not enough, it should be OutOfMemoryError:D irect buffermemory, you can look at Java.nio. Directbytebuffer This kind of source code, 98 lines

Iv. Improvement of NIO2.0

NIO's DMA performance is certainly much better than allocating in the heap, because it is the direct operation of local memory, which avoids the copying of data between jvmheap and local memory, especially when the amount of data is large.

The Java.lang.OutOfMemoryError of Mina