Analysis of Three Tomcat memory overflow conditions and Solutions

Causes of Tomcat memory overflow
In the production environment, if Tomcat memory is not set properly, memory overflow may easily occur. Memory overflow is caused by different processing methods.
Here we will summarize the situation and relevant information. There are three common cases:
1. outofmemoryerror: Java heap Space
2. outofmemoryerror: permgen Space
3. outofmemoryerror: Unable to create new Native thread.
Tomcat memory overflow Solution
In the first two cases, you can set the Tomcat JVM parameter when the application does not leak memory. (-XMS-xmx-XX: permsize-XX: maxpermsize)
The last possible reason is to adjust the operating system and tomcat JVM parameters at the same time.
First: heap overflow.
Cause analysis:
JVM heap setting refers to the setting of memory space that JVM can allocate during Java program running. the JVM automatically sets the heap size value at startup. The initial space (-XMS) is 1/64 of the physical memory, and the maximum space (-xmx) is 1/4 of the physical memory. You can use options such as-xmn-XMS-xmx provided by JVM to set the configuration. The heap size is the sum of young generation and tenured generaion.
This exception is thrown if 98% is used for GC and the available heap size is less than 2% in JVM.
The heap size should not exceed 80% of the available physical memory. Generally, the-XMS and-xmx options must be set to the same, and-xmn is the-xmx value of 1/4.
If there is no memory leakage, you can adjust the-XMS-xmx parameter.
-XMS: initial heap size
-Xmx: Maximum heap size
However, the heap size is affected by the following three factors:
1. Restrictions on the data model (32-bit or 64-bit) of the relevant operating system; (32-bit systems are generally limited to Gbps ~ 2G; I tested 2003 m in the 1.6 Server System (physical memory: 4G and 6g, JDK: 1612). The 64-bit operating system has no restrictions on the memory .)
2. Virtual Memory limit available for the system;
3. Physical memory limit available for the system.
You can use the Java-xmx *** M version command to test the heap size. If yes, the JDK version is displayed. If no, an error is returned.
-XMS-xmx is generally configured as the same, for example, set java_opts =-xms1024m-xmx1024m. Its initial space (I .e.-XMS) is 1/64 of the physical memory, and the maximum space (-xmx) it is 1/4 of the physical memory. You can use options such as-xmn-XMS-xmx provided by JVM.
Set
Example:
Java_opts = "-server-xms800m-xmx800m-XX: permsize = 64 m-XX: Maxnewsize= 256 m-XX: maxpermsize = 128 M-djava. AWT. Headless = true"
Java_opts = "-server-xms768m-xmx768m-XX: permsize = 128 M-XX: maxpermsize = 256 m-XX:
Newsize = 192 m-XX: Maxnewsize= 384 M"
Catalina_opts = "-server-xms768m-xmx768m-XX: permsize = 128 M-XX: maxpermsize = 256 m
-XX: newsize = 192 m-XX: Maxnewsize= 384 M "the server has 1 GB memory: java_opts ="-server-xms800m-xmx800m-XX: permsize = 64 m-XX: Maxnewsize= 256 m-XX: maxpermsize = 128 M-djava. AWT. Headless = true"
The server is 64-bit, 2 GB memory: java_opts = '-server-xms1024m-xmx1536m-XX: permsize = 128 M-XX: Maxnewsize= 256 m-XX: maxpermsize = 256m '--------------------- solution 1 :-----------------------------
Prerequisite: Execute startup. BAT to start Tomcat.
Linux Server:
Catalina. Sh in the/usr/local/Apache-Tomcat-5.5.23/bin directory
Add: java_opts = '-xms512m-xmx1024m'
Or java_opts = "-server-xms800m-xmx800m-XX: Maxnewsize= 256 m"
Or catalina_opts = "-server-xms256m-xmx300m"
Windows Server:
Add at the beginning of Catalina. bat
Set java_opts =-xms128m-xmx350m
Or set catalina_opts =-xmx300m-xms256m
(The difference is that one sets the JVM memory directly, and the other sets the Tomcat memory. catalina_opts seems to be available for use with java_opts in a different area)
Basic parameter description
-Client,-Server
These two parameters are used to set the running mode used by the virtual machine. They must be the first parameter. The client mode is faster to start, but the runtime performance and memory management efficiency are not as good as the server mode, it is usually used in client applications. On the contrary, server mode is slower to start than client, but can achieve higher running performance.
In Windows, the default Virtual Machine Type is client mode. To use server mode, you need to add the-server parameter when starting the virtual machine to achieve higher performance and server-side applications, server mode is recommended, especially for multiple CPU Systems. In Linux, Solaris adopts the server mode by default.
In addition, in multiple CPUs, we recommend that you use server mode-XMS <size>
Set the initial size of the available memory heap of the VM. The default unit is byte. The size is an integer multiple of 1024 and must be greater than 1 MB. The available size is K or M (m) to set a large memory size. The initial heap size is 2 MB. "M" indicates that it is MB, otherwise it is kb.
Example:-xms6400k,-xms256m
-Xmx <size>
Sets the maximum available size of the VM. The default unit is byte. The value must be an integer multiple of 1024 and greater than 2 MB. You can set a large memory size in the unit of K (k) or M (m. The maximum heap size is 64 MB by default.
Example:-xmx81920k,-xmx80m
When the application requests a large memory runtime, the VM throws a java. Lang. outofmemoryerror: Java heap space error, you need to use-xmx to set a large available memory heap.
Permsize/maxpermsize: defines the size of the perm segment, that is, the size of the permanent storage area. permsize indicates the memory size of the perm initialized during JVM startup. maxpermsize indicates the maximum memory size of the perm. In the user production environment, these two values are generally set to the same, to reduce the overhead of the system in the memory application during running. If you start Tomcat with startup. bat, the OK setting takes effect. The memory is allocated MB.
------------------- Solution 2 :------------------------
Prerequisite: Execute startup. BAT to start Tomcat.
Manually set heap size
Windows Server:
Modify tomcat_home/bin/Catalina. BAT and add the following lines to "Echo" using catalina_base: $ catalina_base:
Java code
Set java_opts = % java_opts %-server-xms800m-xmx800m-XX: Maxnewsize= 256 m

Note: java_opts retains the previous settings.
Linux Server:
Modify tomcat_home/bin/Catalina. Sh
Add the following lines to "Echo" using catalina_base: $ catalina_base:
Java_opts = "$ java_opts-server-xms800m-xmx800m-XX: Maxnewsize= 256 m "NOTE: $ java_opts retains the previous settings.
------------------- Solution 3 :-----------------------------
Prerequisite: You can run Windows system services to start Tomcat.
However, if you do not execute startup. BAT to start Tomcat, but use the Windows System Service to start the Tomcat service, the above settings will not take effect,
That is to say, set java_opts =-xms128m-xmx350m does not work. The above allocated M memory will become OOM ..
The Windows service executes bin \ tomcat.exe. It reads the value in the registry, instead of setting Catalina. bat. solution:
Modify the Registry HKEY_LOCAL_MACHINE \ SOFTWARE \ Apache Software Foundation \ Tomcat Service Manager \ tomcat5 \ Parameters \ javaoptions
Original Value:
-Dcatalina. Home = "C: \ apachegroup \ Tomcat 5.0"
-Djava. endorsed. dirs = "C: \ apachegroup \ Tomcat 5.0 \ common \ endorsed"
-Xrs
Add-xms300m-xmx350m
Restart the Tomcat service and the settings take effect.
------------------- Solution 4 :-----------------------------
Prerequisite: You can run Windows system services to start Tomcat.
If you select "NT Service (NT/2000/XP only)" When installing Tomcat )"
After the installation is complete, there will be a atat.exe file in the "bin" directory of the installation directory.
Stop Tomcat services first
In command column mode (run cmd)
Switch the directory to the bin directory of Tomcat
Run the following command to remove the service:
 
Tomcat-Uninstall "Apache Tomcat 4.1"
 
Next, write a batch.
The content is as follows:
Set servicename = Apache Tomcat 4.1
Set catalina_home = E: \ Tomcat 4.1.24
Set classpath = D: \ j2sdk1.4.1 _ 01 \ Lib
Set javaclasspath = % classpath %
Set javaclasspath = % javaclasspath % ;? Talina_home % \ bin \ Bootstrap. Jar
Set javaclasspath = % javaclasspath % ;? Talina_home % \ common \ Lib \ servlet. Jar
Set javaclasspath = % javaclasspath %; % java_home % \ Lib \ tools. Jar
Tomcat.exe-install "% servicename %" "% java_home % \ JRE \ bin \ Server \ JVM. DLL "-djava. class. path = "% javaclasspath %"-dcatalina. home = "? Talina_home % "-xms512m-xmx768m-start org. Apache. Catalina. startup. Bootstrap-Params start-stop org. Apache. Catalina. startup. Bootstrap-Params stop-out "? Talina_home % \ logs \ stdout. log "-Err "? Talina_home % \ logs \ stderr. log"
 
Note that the last line starts with atat.exe-install! Do not manually press enter to wrap the line into several segments. Save and execute the BAT file in the command line. Note that the "service" window is closed during execution. Type 2: Permanently saved region Overflow
Cause analysis:
The full name of permgen space is permanent generation space, which refers to the permanent storage area of the memory. This memory is mainly used by JVM to store class and meta information, when a class is loaded, it will be placed in the permgen space. It is different from the heap region where the class instance is stored. GC (garbage collection) permgen space is not cleaned up during the main program running period. Therefore, if your application has a class, the permgen space error may occur, this error is common when the web server pre-compile the JSP. If your web app uses a large number of third-party jar files and the size exceeds the default JVM size (4 MB), this error message is generated. However, the current Hibernate and spring projects are prone to such problems. It may be because these frameworks will dynamically class and the jvm gc will not clean up the pemgen space, exceeding the default JVM size (4 MB), leading to memory overflow.
Suggestion: Move the same third-party jar files to the tomcat/shared/lib directory to reduce the memory usage of jar files.
This is generally solved by increasing-XX: permsize-XX: maxpermsize.
-XX: permsize: Permanent storage area initial size
-XX: the initial maximum value of the permsize permanent storage area.
This is generally used in combination with the first one, such as set java_opts =-xms1024m-xmx1024m-XX: permsize = 128 M-XX: permsize = 256 m
Note that the maximum heap memory tested by the Java-xmx *** M version command is-xmx and-XX: permsize and, for example, the maximum JVM heap size supported by the system is 1.5 GB, -xmx1024m-XX: permsize = 768 m cannot be run.
----------------- Solution 1 :-------------------------
Linux Server:
Add the following in the first line of Catalina. sh:
Java_opts =
-Xms64m
-Xmx256m
-XX: permsize = 128 m
-XX: Maxnewsize= 256 m
-XX: maxpermsize = 256 m
Or
Add the following lines to "Echo" using catalina_base: $ catalina_base:
Java_opts = "-server-XX: permsize = 64 m-XX: maxpermsize = 128 m
Windows Server:
Add the following in the first line of Catalina. BAT:
Set java_opts =-xms64m-xmx256m-XX: permsize = 128 M-XX: Maxnewsize= 256 m-XX: maxpermsize = 256 m
----------------- Solution 2 :------------------------
Modify tomcat_home/bin/Catalina. BAT (Catalina. Sh in Linux ).
Add the following lines to "Echo" using catalina_base: $ catalina_base:
Set java_opts = % java_opts %-server-XX: permsize = 128 M-XX: maxpermsize = 512 M "Echo" using catalina_base: $ catalina_base "Add the following lines above:
Set java_opts = % java_opts %-server-XX: permsize = 128 M-XX: maxpermsize = 512mcatalina. sh:
Java code
Java_opts = "$ java_opts-server-XX: permsize = 128 M-XX: maxpermsize = 512 M" java_opts = "$ java_opts-server-XX: permsize = 128 M-XX: maxpermsize = 512 M "3: A New thread cannot be created.
This phenomenon is rare and strange, mainly due to the ratio of JVM to system memory.
This is strange because the JVM has been allocated a large amount of memory (such as 1.5 GB) by the system, and it must occupy at least half of the available memory. Some people find that the more memory you allocate to the JVM when there are many threads, the more likely the above errors will occur.
Cause Analysis
(Learn the cause from this blog: http://hi.baidu.com/hexiong/blog/item/16dc9e518fb10c2542a75b3c.html ):
Each 32-bit process can use up to 2 GB of available memory, because the other 2 GB is retained by the operating system. Assume that GB memory is used for JVM, then MB of memory is available. The part of the 400 MB memory must be used for system DLL loading, and the rest may only be mb. Now the key point is: When you use Java to create a thread, A thread object will also be created in the JVM memory, but a real physical thread will also be created in the operating system (refer to JVM specifications ), the operating system will create this physical thread in the remaining 400 mb of memory, instead of creating it in the JVM's MB memory heap. In JDK, the default stack size is kb, but in JDK, the default stack size is 1 MB per Thread. Therefore, up to 400 available threads can be created in the remaining MB of available memory.
This is the conclusion. to create more threads, you must reduce the maximum memory allocated to the JVM. Another way is to host the JVM in your JNI code.
An Estimation Formula for the maximum number of threads that can be created is provided:
(Maxprocessmemory-jvmmemory-reservedosmemory)/(threadstacksize) = number of threads
For JDK, assume that the operating system retains MB of memory:
1.5 gb jvm: (2gb-1.5gb-120mb)/(1 MB) = ~ 380 threads
1.0 gb jvm: (2gb-1.0gb-120mb)/(1 MB) = ~ 880 threads
In 2000/XP/2003 boot. there is a startup option in ini, which seems to be:/PAE/3G, which allows the user process to expand the maximum memory to 3G. In this case, the operating system can only occupy up to 1 GB of virtual memory. In this way, the JVM should be able to create more threads.
Therefore, adjustments must be made based on the operating system.
Therefore, we need to diagnose Tomcat memory allocation based on different situations to fundamentally solve the problem.
 
Check the current JVM memory usage:
System. Out. println ("JVM Max memory:" + runtime. getruntime (). maxmemory ()/1024/1024 + "M ");
System. Out. println ("JVM is using memory:" + runtime. getruntime (). totalmemory ()/1024/1024 + "M ");
System. out. println ("JVM is free memory:" + runtime. getruntime (). freememory ()/1024/1024 + "M"); all three methods are about JVM memory usage rather than the operating system memory;
The maxmemory () method returns the maximum memory that the Java Virtual Machine (this process) can dig from the operating system, in bytes. If you are running a Java program, if the-xmx parameter is not added, it is 64 mb. That is to say, maxmemory () returns about 64*1024*1024 bytes, this is the maximum memory that the Java Virtual Machine can dig from the operating system by default. If the-xmx parameter is added, the value following this parameter prevails. For example, Java-CP classpath-xmx512m classname, the maximum memory size is 512*1024*0124 bytes.
 
The totalmemory () method returns the memory size that the Java Virtual Machine has dug from the operating system, that is, all the memory occupied by the Java Virtual Machine process at that time. If the-XMS parameter is not added when running Java, the memory is always slowly dug from the operating system during the Java program running process, basically how much is used to dig, until maxmemory (), totalmemory () increases slowly. If the-XMS parameter is used, the program will unconditionally dig the memory size defined after-XMS from the operating system at startup, and then when the memory usage is similar, then dig.
 
What is freememory ()? If the-XMS parameter is not added while running Java, the process of running Java is as follows, the memory is always dug slowly from the operating system, basically the number of resources to be dug, but the Java Virtual Machine will dig a little more in 100% cases, the memory is actually freememory (), so the value of freememory () is usually very small, however, if you use-XMS when running a Java program, this is because the program will unconditionally dig out the number of memory defined after the XMS from the operating system at startup, at this time, most of the dug memory may be useless, so freememory () may be somewhat
-------------------- Solution --------------------------
JVM heap size adjustment
Sun hotspot 1.4.1 uses the generational collector, which divides the heap into three main domains: new domain, old domain, and permanent domain. All new objects generated by JVM are placed in the new domain. Once an object goes through a certain number of garbage collection cycles, it will get the validity period and enter the old domain. In the permanent domain, JVM stores class and method objects. In terms of configuration, the permanent domain is an independent domain and is not considered a part of the heap.
The following describes how to control the size of these domains. You can use-XMS and-xmx to control the original size or maximum value of the entire heap.
The following command sets the initial size to 128 MB:
Java-xms128m
-Xmx256m: to control the size of the new domain, you can use-XX: newratio to set the proportion of the new domain in the heap.
The following command sets the entire heap to 128 m, and the ratio of the new domain to 3, that is, the ratio of the new domain to the old domain is, and the new domain is 1/4 or 32 m of the heap:
Java-xms128m-xmx128m
-XX: newratio = 3-XX: newsize and-XX: MaxnewsizeSet the initial value and maximum value of the new domain.
The following command sets the initial value and maximum value of the new domain to 64 MB:
Java-xms256m-xmx256m-xmn64m
The default size of the permanent domain is 4 MB. When the program is running, the JVM will adjust the size of the permanent domain to meet the needs. During each adjustment, the JVM will perform a full garbage collection on the heap.
Use the-XX: maxpersize flag to increase the size of the permanent domain. When WebLogic Server Applications load many classes, you often need to increase the maximum value of the permanent domain. When the JVM loads a class, the object in the permanent domain increases sharply, so that the JVM continuously adjusts the permanent domain size. To avoid adjustment, you can use the-XX: persize flag to set the initial value.
Set the initial value of the permanent domain to 32 MB and the maximum value to 64 MB.
Java-xms512m-xmx512m-xmn128m-XX: permsize = 32 m-XX: maxpermsize = 64 m
By default, hotspot uses the replication collector in the new domain. This domain is generally divided into three parts. The first part is Eden, which is used to generate new objects. The other two parts are called Rescue space. When Eden is full, the collector stops the application and copies all reachable objects to the current from rescue space. Once the current from rescue space is full, the collector copies the reachable objects to the current to rescue space. From and to rescue space swap roles. Objects that maintain the activity will be continuously replicated in the rescue space until they are used and transferred to the old domain. -XX: survivorratio can be used to control the size of the new domain sub-space.
Like newration, mirror vorration specifies the ratio of a rescue domain to an Eden space. For example, the following command sets the new domain to 64 m, and Eden to 32 m, each of which occupies 16 m:
Java-xms256m-xmx256m-xmn64m-XX: Required vorration = 2
As mentioned above, by default, hotspot uses the replication collector for the new domain and the tag-clear-compression collector for the old domain. Using the replication collector in the new domain makes a lot of sense, because most of the objects generated by the application are short-lived. Ideally, all transition objects will be collected when they are removed from the Eden space. If this is the case, and objects removed from the Eden space are long-lived, then theoretically they can be immediately moved to the old domain to avoid repeated replication in the rescue space. However, applications cannot fit this ideal state because they have a small portion of long-lived objects. It is best to keep these long-lived objects in the new domain, because copying a small part of the object is always cheaper than compressing the old domain. To control the replication of objects in the new domain, you can use-XX: target1_vorratio to control the proportion of the rescue space (this value is used to set the proportion of the rescue space. For example, if the rescue space is 1 MB, the value 50 indicates 500 k is available ). This value is a percentage and the default value is 50. When a larger stack uses a lower sruvivorratio value, the value should be increased to 80 to 90 to make better use of the rescue space. -XX: maxtenuring threshold can control the upper limit.
To place all copies and extend the objects from Eden to the old domain, you can set maxtenuring threshold to 0. After the configuration is complete, the rescue space is no longer used. Therefore, set the maximum vorratio value to maximize the Eden space. The settings are as follows:
Java... -XX: maxtenuringthreshold = 0-XX: Export vorratio = 50000...
Description of garbage collection:
Garbage collection is divided into multiple levels. garbage collection in the old section will be collected when the level 0 is full. garbage collection in the Level 1 or above will only be collected from young, memory overflow usually occurs after garbage collection of the old or perm segments, and there is still no memory space to accommodate new Java objects.
When a URL is accessed, the memory application process is as follows:
A. JVM will try to initialize a memory area for the relevant Java object in Eden
B. When the Eden space is sufficient, the memory application is completed. Otherwise, go to the next step.
C. JVM tries to release all inactive objects in Eden (this belongs to 1 or more advanced garbage collection); If the Eden space is not enough to put new objects after the release, try to put some active objects in Eden into region VOR/old
D. The primary vor area is used as the intermediate swap area of Eden and old. When the old area has sufficient space, objects in the primary vor area will be moved to the old area. Otherwise, objects in the primary vor area will be retained.
E. When there is not enough space in the old area, JVM will perform full garbage collection in the old area (Level 0)
F. after full garbage collection, if the primary VOR and old areas still cannot store some objects copied from Eden, the JVM cannot create a memory area for the new object in the Eden area, "Out of memory error" appears"
Java heap parameters:
MS/MX: defines the total size of the Young + old segment, Ms is the size of the Young + old memory at JVM startup, and MX is the maximum size of the Young + old memory that can be occupied. In the user production environment, these two values are generally set to the same, to reduce the overhead of the system in the memory application during running.
Newsize/ Maxnewsize: Defines the size of the young segment. newsize is the memory size of young at JVM startup; MaxnewsizeIs the maximum memory size that can be occupied by young. In the user production environment, these two values are generally set to the same, to reduce the overhead of the system in the memory application during running.
Permsize/maxpermsize: defines the size of the perm segment. permsize indicates the memory size of the perm when the JVM starts. maxpermsize indicates the maximum available perm memory size. In the user production environment, these two values are generally set to the same, to reduce the overhead of the system in the memory application during running.
Survivorratio: Set the ratio of VOR space to Eden space.
Example:
Mem_args = "-xms512m-x mx512m-XX: newsize = 256 m-XX: Maxnewsize= 256 m-XX: permsize = 128 M-XX: maxpermsize = 128 M-XX: Export vorratio = 6"
In the above example:
Young + Old: 512 m
YOUNG: 256 m
Perm: 128 m
EDEN: Young * 6/(6 + 1 + 1) = 192 m
Required vor: Young/(6 + 1 + 1) = 32 m
Java heap size = Young + Old + perm = 640 m

Analysis of Three Tomcat memory overflow conditions and Solutions