Dead locks that disappear

dead locks that disappearProblem description

If there is a deadlock at the Java level, when we use the Jstack command, we can actually dump the deadlock information, and at the end of the dump there will be a keyword like found one java-level deadlock: Then the deadlock occurs on the thread's stack and corresponding synchronization lock to print out, this time a system encountered a similar problem, but this dump document, although the following deadlock information is mentioned:

Found One java-level deadlock:
=============================
"worker-1-thread-121":
Waiting to lock monitor 0x00007f3758209dc8 (Object 0x0000000764cd2b20, a java.util.concurrent.ConcurrentHashMap),
Which is held by "hsfbizprocessor-4-thread-4"
"Hsfbizprocessor-4-thread-4":
Waiting to lock monitor 0x00007f3758289260 (object 0x000000076073ddc8, a Com.rjb.test.extensions.equinox.KernelBundleClassLoader),
Which is held by "hsfbizprocessor-4-thread-5"
"Hsfbizprocessor-4-thread-5":
Waiting to lock monitor 0x00007f3758253420 (object 0x00000007608e6fc8, a Com.rjb.test.extensions.equinox.KernelBundleClassLoader),
Which is held by "hsfbizprocessor-4-thread-4"

But when we searched for the corresponding lock in the stack, we didn't find it, which is the above mentioned

Object 0x00000007608e6fc8 which is held by "hsfbizprocessor-4-thread-4"

We do not see the corresponding lock information in the stack of the hsfbizprocessor-4-thread-4 thread.

Attached to the online dump details

Found One java-level deadlock: ============================= "worker-1-thread-121":  waiting to lock monitor 0x0000 7f3758209dc8 (Object 0x0000000764cd2b20, a java.util.concurrent.ConcurrentHashMap),  which is held by " Hsfbizprocessor-4-thread-4 "hsfbizprocessor-4-thread-4":  waiting to lock monitor 0x00007f3758289260 (object 0x000000076073ddc8, a com.rjb.test.extensions.equinox.KernelBundleClassLoader),  which is held by " Hsfbizprocessor-4-thread-5 "hsfbizprocessor-4-thread-5":  waiting to lock monitor 0x00007f3758253420 (object 0x00000007608e6fc8, a com.rjb.test.extensions.equinox.KernelBundleClassLoader),  which is held by " Hsfbizprocessor-4-thread-4 "Java stack information for the threads listed above: ====================================== ============= "worker-1-thread-121":    at Org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton ( defaultsingletonbeanregistry.java:180)    -waiting to lock <0x0000000764cd2b20> (a java.util.concurrent.ConcurrentHashMap)    at Org.springframework.beans.factory.support.AbstractBeanFactory.isTypeMatch (abstractbeanfactory.java:455)    at Org.springframework.beans.factory.support.DefaultListableBeanFactory.getBeanNamesForType ( defaultlistablebeanfactory.java:317)   &nbsp, ...    at Java.util.concurrent.FutureTask.run ( futuretask.java:138)    at Java.util.concurrent.threadpoolexecutor$worker.runtask ( threadpoolexecutor.java:886)    at Java.util.concurrent.threadpoolexecutor$worker.run (
 threadpoolexecutor.java:908)    at java.lang.Thread.run (thread.java:662) "hsfbizprocessor-4-thread-4":    at Org.eclipse.osgi.baseadaptor.loader.ClasspathManager.findLoadedClass (Unknown Source)    - Waiting to lock <0x000000076073ddc8> (a com.rjb.test.extensions.equinox.KernelBundleClassLoader)     At Org.eclipse.osgi.baseadaptor.loader.ClasspathManaGer.findlocalclass (Unknown Source)    at Org.eclipse.osgi.internal.baseadaptor.DefaultClassLoader.findLocalClass (Unknown Source)    at Org.eclipse.osgi.internal.loader.BundleLoader.findLocalClass (Unknown Source)    at Org.eclipse.osgi.internal.loader.SingleSourcePackage.loadClass (Unknown Source)    at Org.eclipse.osgi.internal.loader.BundleLoader.findClassInternal (Unknown Source)    at Org.eclipse.osgi.internal.loader.BundleLoader.findClass (Unknown Source)    at Org.eclipse.osgi.internal.loader.BundleLoader.findClass (Unknown Source)    at Org.eclipse.osgi.internal.baseadaptor.DefaultClassLoader.loadClass (Unknown Source)    at Com.rjb.test.extensions.equinox.KernelBundleClassLoader.loadClass (kernelbundleclassloader.java:121)    at Java.lang.ClassLoader.loadClass (classloader.java:247)    at Org.springframework.scripting.groovy.GroovyScriptFactory.executeScript (groovyscriptfactory. java:238)   &nbsp, ...    at java.util.concurrent.threadpoolexecutor$worker.runtask ( threadpoolexecutor.java:886)    at Java.util.concurrent.threadpoolexecutor$worker.run (
 threadpoolexecutor.java:908)    at java.lang.Thread.run (thread.java:662) "hsfbizprocessor-4-thread-5":    at Org.eclipse.osgi.baseadaptor.loader.ClasspathManager.findLoadedClass (Unknown Source)    - Waiting to lock <0x00000007608e6fc8> (a com.rjb.test.extensions.equinox.KernelBundleClassLoader)     At Org.eclipse.osgi.baseadaptor.loader.ClasspathManager.findLocalClass (Unknown Source)    at Org.eclipse.osgi.internal.baseadaptor.DefaultClassLoader.findLocalClass (Unknown Source)    at Org.eclipse.osgi.internal.loader.BundleLoader.findLocalClass (Unknown Source)    at Org.eclipse.osgi.internal.loader.BundleLoader.findClassInternal (Unknown Source)    at Org.eclipse.osgi.internal.loader.BundleLoader.findClass (Unknown source)    at Org.eclipse.osgi.internal.loader.BundleLoader.findClass (Unknown source)    at Org.eclipse.osgi.internal.loader.buddy.DependentPolicy.loadClass (Unknown Source)    at Org.eclipse.osgi.internal.loader.buddy.PolicyHandler.doBuddyClassLoading (Unknown Source)    at Org.eclipse.osgi.internal.loader.BundleLoader.findClassInternal (Unknown Source)    at Org.eclipse.osgi.internal.loader.BundleLoader.findClass (Unknown Source)    at Org.eclipse.osgi.internal.loader.BundleLoader.findClass (Unknown Source)    at Org.eclipse.osgi.internal.baseadaptor.DefaultClassLoader.loadClass (Unknown Source)    at Com.rjb.test.extensions.equinox.KernelBundleClassLoader.loadClass (kernelbundleclassloader.java:121)    at Java.lang.ClassLoader.loadClass (classloader.java:247)    at JAVA.LANG.CLASS.FORNAME0 (Native Method)    at java.lang.Class.forName (class.java:169)   at Groovy.lang.metaclassregistry$metaclasscreationhandle.createwithcustomlookup (MetaClassRegistry.java
 : 127)    at groovy.lang.metaclassregistry$metaclasscreationhandle.create (metaclassregistry.java:122)   &nbsp, ...    at java.util.concurrent.threadpoolexecutor$worker.runtask (Threadpoolexecutor.java : 886)    at Java.util.concurrent.threadpoolexecutor$worker.run (threadpoolexecutor.java:908)     At Java.lang.Thread.run (thread.java:662) Found 1 deadlock.

The problem with class loading.

The problem of class loading should be suspected first, because we see that the object causing the deadlock is a ClassLoader object:

Waiting to lock monitor 0x00007f3758289260 (object 0x000000076073ddc8, a Com.rjb.test.extensions.equinox.KernelBundleClassLoader)

And then we'll analyze the stack hsfbizprocessor-4-thread-4 .

"Hsfbizprocessor-4-thread-4":    at Org.eclipse.osgi.baseadaptor.loader.ClasspathManager.findLoadedClass (Unknown Source)    -waiting to lock <0x000000076073ddc8> (a com.rjb.test.extensions.equinox.KernelBundleClassLoader)    at Org.eclipse.osgi.baseadaptor.loader.ClasspathManager.findLocalClass (Unknown Source)    at Org.eclipse.osgi.internal.baseadaptor.DefaultClassLoader.findLocalClass (Unknown Source)    at Org.eclipse.osgi.internal.loader.BundleLoader.findLocalClass (Unknown Source)    at Org.eclipse.osgi.internal.loader.SingleSourcePackage.loadClass (Unknown Source)    at Org.eclipse.osgi.internal.loader.BundleLoader.findClassInternal (Unknown Source)    at Org.eclipse.osgi.internal.loader.BundleLoader.findClass (Unknown Source)    at Org.eclipse.osgi.internal.loader.BundleLoader.findClass (Unknown Source)    at Org.eclipse.osgi.internal.baseadaptor.DefaultClassLoader.loadClass (Unknown Source)    at Com.rjb.test.extensions.equinox.KernelBundleClassLoader.loadClass ( kernelbundleclassloader.java:121)    at Java.lang.ClassLoader.loadClass (classloader.java:247)  
  at Org.springframework.scripting.groovy.GroovyScriptFactory.executeScript (groovyscriptfactory.java:238)    at Org.springframework.scripting.groovy.GroovyScriptFactory.getScriptedObject ( groovyscriptfactory.java:185)

I'm just going to stick the key thread stack here and know that it's waiting for a lock from the top of the stack:

-Waiting to lock <0x000000076073ddc8> (a com.rjb.test.extensions.equinox.KernelBundleClassLoader)

This lock object is a ClassLoader object, we find the corresponding code, there is really synchronized operation:

Private class<?> Findloadedclass (String classname) {
if ((lock_classname) | | | (This.isparallelclassloader)) {
Boolean initiallock = Lockclassname (classname);
try {
return this.classloader.publicFindLoaded (classname);
} finally {
if (Initiallock)
Unlockclassname (classname);
}
}
Synchronized (This.classloader) {
return this.classloader.publicFindLoaded (classname);
}
}

We also know that it is performing the loadclass action, and it is called from groovy, and the corresponding code is also found:

Protected Object Executescript (Scriptsource scriptsource, Class scriptclass)
Throws Scriptcompilationexception
{
Try
{
Groovyobject goo = (groovyobject) scriptclass.newinstance ();//line 238

if (This.groovyobjectcustomizer! = null)
{
This.groovyObjectCustomizer.customize (goo);
}

if ((Goo instanceof Script))
{
Return (Script) goo. Run ();
}

return goo;
}
catch (Instantiationexception ex)
{
throw New Scriptcompilationexception (
Scriptsource, "Could not instantiate Groovy script class:" + Scriptclass.getname (), ex);
}
catch (Illegalaccessexception ex) {
throw New Scriptcompilationexception (
Scriptsource, "Could not access Groovy script constructor:" + Scriptclass.getname (), ex);
}
}

When we go to line No. 238,

Groovyobject goo = (groovyobject) scriptclass.newinstance ();//line 238

A sudden discovery called the

Java.lang.ClassLoader.loadClass (classloader.java:247)

And we see that the logic in line No. 238 above is actually instantiating an object and then doing a strong turn, and we'll look at the corresponding byte code:

0:aload_2
 1:invokevirtual #164//Method java/lang/class.newinstance: () ljava/lang/object;
 4:checkcast #168//class Groovy/lang/groovyobject
 7:astore_3

In fact, it corresponds to such a few bytecode instructions, in fact, in the JVM when we execute the checkcast instruction will trigger the class loading action:

void Templatetable::checkcast () {
...
CALL_VM (Rax, cast_from_fn_ptr (address, interpreterruntime::quicken_io_cc));
...
}

Irt_entry (void, INTERPRETERRUNTIME::QUICKEN_IO_CC (javathread* thread))
Force resolving; Quicken the bytecode
int which = GET_INDEX_U2 (thread, bytecodes::_checkcast);
Constantpooloop Cpool = Method (thread)->constants ();
We ' d expect to asserts that we ' re is only here to Quicken Bytecodes, but in a multithreaded
Program we might has seen an Unquick ' d bytecode in the interpreter but has another
Thread Quicken the bytecode before we get here.
ASSERT (Cpool->tag_at (which). Is_unresolved_klass (), "should only come here to Quicken Bytecodes");
Klassoop Klass = Cpool->klass_at (which, CHECK);
Thread->set_vm_result (Klass);
Irt_end

Klassoop klass_at (int which, TRAPS) {
Constantpoolhandle H_this (THREAD, this);
Return Klass_at_impl (H_this, which, check_null);
}

Klassoop Constantpooloopdesc::klass_at_impl (constantpoolhandle this_oop, int which, TRAPS) {
...
Klassoop k_oop = systemdictionary::resolve_or_fail (name, loader, H_prot, True, THREAD);
...
}

Systemdictionary::resolve_or_fail will eventually be called to the following method
Klassoop Systemdictionary::resolve_instance_class_or_null (symbol* name, Handle class_loader, Handle protection_ Domain, TRAPS) {
...