Weblogic OutOfMemory Exception's misunderstanding--thread limitation

Not all OutOfMemory exception are memory problems ... A few days ago a customer's site reported the following error:

[ERROR] [Thread] Could not start thread Timer-72025. Resource temporarily unavailableexception in thread "Timer-72024" Java.lang.OutOfMemoryError:Resource temporarily Unavailable in Tsstartjavathread (lifecycle.c:1097). Attempting to allocate 2G bytesthere are insufficient native memory for the javaruntime environment to continue.  Possible reasons:the System is out of physical RAM or swap space in + bit mode, the process size limit was hitpossible   Solutions:reduce memory load on the system increase physical memory or swap space Check if swap backing store are full Use + bit Java on a decrease Java heap size (-XMX/-XMS) Decrease number of Java threads decrease Java thre Ad stack Sizes (-XSS) Disable compressed references (-xxcompressedrefs=false) at Java.lang.Thread.start0 (N Ative Method) at Java.lang.Thread.start (thread.java:640) at java.util.timer.<init> (Tim er.java:137) at java.util.timer.<init>(timer.java:106) at Java.util.TimerThread.mainLoop (timer.java:512) At Java.util.TimerThread.run (timer.java:462)

The first reaction is that the memory is not enough, so dump the heap back to see that the system only occupies 80M of memory, far more than the system memory, then why this situation? Looking at the code that went wrong, and discovering that the code started a lot of threads, I thought of a similar problem that had been encountered before, WebLogic the thread that started too much, exceeded the maximum limit of Linux, and the log that looked like a memory leak. To view system limitations:

[Email protected] LIMITS.D # ulimit-acore file size          (blocks,-c) 0data seg size           (Kbytes,-D) unlimitedscheduling Priority             (-e) 0file size               (blocks,-f) unlimitedpending signals (-I                 ) 38610max locked memory       (Kbytes, L) 64max Memory Size         (Kbytes,-m) unlimitedopen files                      (-N) 8192pipe size            (bytes,-p) 8POSIX message Queues
    (bytes,-Q) 819200real-time priority              (-R) 0stack size              (Kbytes,-s) 10240cpu time               (seconds,-t) Unlimitedmax User Processes              (-u) 1024virtual memory          (Kbytes,-V) unlimitedfile locks                      (-X) Unlimited

Discover that the largest number of processes is 1024, note: Linux inside the compute threads and processes are calculated in the same way, the max user processes that is included here is the thread.

Use the following code to reproduce the problem:

public class Threadexample {public    static void Main (string[] args) {        System.out.println (Thread.CurrentThread ( ). GetName ());        for (int i=0, i< 2048; i++) {            new Thread ("" + i) {public                void run () {                    System.out.println ( Thread.CurrentThread (). GetName () + "Running");                    try {                        thread.sleep (60000);                    } catch (Interruptedexception e) {                        e.printstacktrace ();                    }                    System.out.println (Thread.CurrentThread (). GetName () + "ending");                }            }. Start ();}}    }

Execution and output:

[Email protected]_z1 Etmdaha # su-p occas-c "/opt/jrockit-jdk/bin/java threadexample" Main Thread0 running5 Running 836 Running[error][thread] Could not start thread 837. Resource temporarily unavailableexception in thread "Main thread" Java.lang.OutOfMemoryError:Resource temporarily Unavailable in Tsstartjavathread (lifecycle.c:1096). Attempting to allocate 3G bytesthere are insufficient native memory for the javaruntime environment to continue.  Possible reasons:the System is out of physical RAM or swap space in + bit mode, the process size limit was hitpossible  Solutions: <strong>reduce memory load on the system increase physical memory or swap space Check if swap backing Store is full use # bit Java on a decrease bit OS java heap size (-XMX/-XMS) Decrease number of Java threads DECR Ease Java thread stack sizes (-XSS) Disable compressed references (-xxcompressedrefs=false) </strong> at Java . lang. Thread.start0 (Native Method) at Java.lang.Thread.Start (thread.java:640) at Threadexample.main (Threadexample.java:6) 

Linux system resources have various limitations: memory limit, stack limit, number of files limit, thread limit, to be careful to trap these problems.

# ulimit-acore File size          (blocks,-c) 0data seg size           (Kbytes,-D) unlimitedscheduling priority             (-e) 0file siz e               (blocks,-f) unlimitedpending signals (-I                 ) 79010max locked memory       (Kbytes, L) 64max memory size         ( Kbytes,-m) unlimitedopen files                      (-N) 1024pipe size            (bytes,-p) 8POSIX message Queues     (bytes,-Q) 819200r Eal-time Priority              (-R) 0stack size              (Kbytes,-s) 10240cpu time               (seconds,-t) Unlimitedmax user processes< c15/> (-u) 1024virtual memory          (Kbytes,-V) unlimitedfile locks                      (-X) Unlimited

Here are two very good articles:

Http://people.redhat.com/alikins/system_tuning.html#threads

http://kavassalis.com/2011/03/linux-and-the-maximum-number-of-processes-threads/