EJB and Java Serialization (1) Discovering and analyzing problems

This is the first project to take over after joining the new company, using weblogic9.2 + ejb2.0, the stress test found that the speed is very slow, the response time is very unsatisfactory, the check log found that some of the EJB invoke the method call time is very long, up to 300-500 milliseconds. Very exaggerated, because the two logs are just an interval between an EJB call. After a thread dump analysis, it was found that a considerable number of threads were waiting, and checking the thread call to discover that when the parameter was serialized, the thread tried to lock it but the lock was occupied, so it was in a wait state. Given this moment of thread dump, as many as 30-50 threads are trying to lock the same lock at the same time, it is clear that the lock is very competitive.

Therefore, it is strongly suspected that the Java serialization mechanism caused the problem, because weblogic/ejb such as too complex inconvenient test, so write a separate class to simulate this scenario:
1. There are a large number of threads running, during which there are serialized operations
2. The serialized object is larger, with a large number of child objects (child objects ...)

Run the test code, the problem recurs, the test opened 50 threads, thread dump and analysis thread dump information, found 49 threads in waiting state, only one thread is working! So, aside from WEBLOGIC/EJB's serialization mechanism for analyzing Java individually, first look at thread dump threads information:

The thread that holds the lock:

"Testthread21" prio=6 tid=0x0ad2d3b8 nid=0x224 runnable [0x0b48f000. 0x0b48fce4]
at Java.io.ObjectStreamClass.processQueue (Unknown Source)
at Java.io.ObjectStreamClass.lookup ( Unknown source)
at Java.io.ObjectOutputStream.writeObject0 (Unknown source)
at Java.io.ObjectOutputStream.writeObject (Unknown source)
at Java.util.ArrayList.writeObject (Unknown source)
At Sun.reflect.GeneratedMethodAccessor1.invoke (Unknown Source)
at Sun.reflect.DelegatingMethodAccessorImpl.invoke (Unknown Source)
at Java.lang.reflect.Method.invoke (Unknown Source)
at Java.io.ObjectStreamClass.invokeWriteObject (Unknown Source)
at Java.io.ObjectOutputStream.writeSerialData (Unknown Source)
at Java.io.ObjectOutputStream.writeOrdinaryObject ( Unknown source)
at Java.io.ObjectOutputStream.writeObject0 (Unknown source)
at Java.io.ObjectOutputStream.writeObject (Unknown Source)
at Test. Test.test (test.java:68)
at Test. Test.run (test.java:53)
at JAVA.LANG.THREad.run (Unknown Source)

One of the threads waiting to be locked: "testthread49" prio=6 tid=0x0ad9a508 nid=0xa9c waiting for monitor entry [0x0bb8f000. 0X0BB8FBE4]
at Java.lang.ref.ReferenceQueue.poll (Unknown Source)
-Waiting to lock <0x02fb1d40> (a Java.lang.ref.referencequeue$lock)
at Java.io.ObjectStreamClass.processQueue (Unknown Source)
at Java.io.ObjectStreamClass.lookup (Unknown source)
at Java.io.ObjectOutputStream.writeObject0 (Unknown source)
at Java.io.ObjectOutputStream.writeObject (Unknown source)
at Java.util.ArrayList.writeObject (Unknown source
at Sun.reflect.GeneratedMethodAccessor1.invoke (Unknown Source)
at Sun.reflect.DelegatingMethodAccessorImpl.invoke (Unknown Source)
at Java.lang.reflect.Method.invoke (Unknown Source)
at Java.io.ObjectStreamClass.invokeWriteObject (Unknown Source)
at Java.io.ObjectOutputStream.writeSerialData (Unknown Source)
at Java.io.ObjectOutputStream.writeOrdinaryObject ( Unknown Source)
at Java.io.ObjectOutputStream.writeObject0 (unknoWN Source
at Java.io.ObjectOutputStream.writeObject (Unknown source)
at Test. Test.test (test.java:68)
at Test. Test.run (test.java:53)
at Java.lang.Thread.run (Unknown Source)

You can see that the problem occurs in the method Processqueue () method in class Java.io.ObjectStreamClass: static void processQueue(ReferenceQueue<Class<?>> queue,
ConcurrentMap<? extends
WeakReference<Class<?>>, ?> map)
{
Reference<? extends Class<?>> ref;
while((ref = queue.poll()) != null) {
map.remove(ref);
}
}

Here the Queue.poll () has a lock and continues to follow poll () code: public Reference<? extends T> poll() {
if(queueEmpty) return null;
synchronized (lock) {
return reallyPoll();
}
}