JVM Bugs: Multiple threads hold a lock

JVM Thread Dump Bug description

In the Java language, when a synchronized block (Synchronized) is accessed concurrently by multiple threads, a heavy-level lock based on mutual exclusion is used in the JVM. The JVM allows only one thread to hold the lock, and if other threads want to acquire the lock, it must be in a waiting state, meaning that when the synchronized block is concurrently accessed, at most there will be a thread in the runnable state holding a lock, The other threads are in a blocked state because they cannot compete for this lock. However, there are times when we find a thread in the blocked state, and the top of the frame prints the lock object it is waiting for, and it also appears to be-locked, which is the same as the result of the thread holding the lock (see figure below). But this is not the case with threads that contrast other blocked states. A different result may occur when we dump the thread again. The test shows that this may be a contingency, and this is a case of a study of the implementation within the JVM to find its root cause.

Jstack the entire process of the command

The above mentioned thread dump, then have to mention the execution thread dump tool---jstack, this tool is a Java self-contained tool, and Java in the same directory, mainly for dump threads, and perhaps everyone also use kill-3 command to dump threads, But the most obvious difference between the two is that the dump content of the former is output by the jstack process, and the target JVM process sends the dump content to the Jstack process (note that this is a scene without the-m parameter, specifying the-m parameter is a bit different, It uses the API of the serviceability agent to implement, the underlying through the ptrace way to get the content of the target process, the execution process may be longer than normal mode, which means that you can do file redirection, the thread dump content output to the specified file , and the latter is output by the target process, will only be generated in the target process of the standard output file, if the standard output itself there is content, it will appear to be messy, such as the use of some analysis tools to analyze, if the tool did not do filtering operations, it is likely to be unable to analyze. So in general we try to use Jstack, and Jstack also has a lot of practical parameters, such as Jstack PID >thread_dump.log, which will dump the thread of the specified PID process into the current directory Thread_ Dump.log file.

Jstack is implemented using Java, which sends a threaddump command to the target JVM process, and the listener thread (Attachlistener) of the target JVM listens to the commands passed in real time ( In fact, the Attachlistener thread is not created as a startup, it is lazy created, and when Attachlistener receives the Threaddump command, the Thread_dump method is invoked to handle the dump operation ( Method in AttachListener.cpp).

Static Jint Thread_dump (attachoperation* op, outputstream* out) {
  bool print_concurrent_locks = false;
  if (op->arg (0)!= NULL && strcmp (op->arg (0), "-l") = = 0) {
    print_concurrent_locks = true;
  }

  Thread Stacks
  vm_printthreads op1 (out, print_concurrent_locks);
  Vmthread::execute (&OP1);

  JNI Global handles
  Vm_printjni OP2 (out);
  Vmthread::execute (&OP2);

  Deadlock detection
  vm_finddeadlocks OP3 (out);
  Vmthread::execute (&OP3);

  return JNI_OK;
}

As you can see from the above method, the Jstack command performs three actions:

Vm_printthreads: Print Thread stacks

Vm_printjni: Print JNI

Vm_finddeadlocks: Print a deadlock

All three operations are given to the vmthread thread to execute, and the Vmthread thread has and will only have one in the entire Java process. Imagine the simple execution of a vmthread thread: Constantly polling a list of tasks and performing tasks sequentially when there are tasks. When a task executes, it will decide whether to suspend the entire application based on the specific task, that is, stop the world, does this remind us of the familiar GC process? Yes, our YGC and CMSGC two stages of application (Init_mark and remark) are executed by this thread, and all require the application to be paused. In fact, the above three operations are required to suspend the entire application, that is, jstack triggered thread dump process will also suspend the application, but this process is generally very soon to end, there will be no obvious feeling. Another memory dump of the JMAP command, will also suspend the entire application, if the use of the-f parameters, the bottom is also using the Serviceability Agent API to dump, but the speed of dump memory is significantly slower.

Vmthread the process of performing a task

The tasks performed by Vmthread are called vm_opration, and there are two vm_opration in the JVM, one that needs to be executed within a security point (the so-called safety point is that the system is in a safe state, except that the vmthread thread can run normally. Other threads must suspend execution, in which case the current series of vm_opration can be safely performed, and the other is not required to be performed within a security point. And this time we're talking about the thread dump that needs to be executed within the security point.

The following is the logic for Vmthread polling:

void Vmthread::loop () {assert (_cur_vm_operation = NULL, "No current one should is executing"); while (true) {...//has acquired a vm_operation if (_cur_vm_operation->evaluate_at_safepoint ()) {//if the Vm_o 
        Peration need to execute _vm_queue->set_drain_list (safepoint_ops) within the safety point;
        Safepointsynchronize::begin ()//Enter the safe point evaluate_operation (_cur_vm_operation);
          do {_cur_vm_operation = Safepoint_ops;
              if (_cur_vm_operation!= NULL) {do {vm_operation* next = _cur_vm_operation->next ();
              _vm_queue->set_drain_list (next);
              Evaluate_operation (_cur_vm_operation);
              _cur_vm_operation = Next;
              if (printsafepointstatistics) {safepointsynchronize::inc_vmop_coalesced_count ();
          } while (_cur_vm_operation!= NULL); } if (_vm_queue->peek_at_safepoint_priority ()) {Mutexlockerex Mu_queue (Vmoperationqueue_lock, Mutex::_no_safepoint_check_flag);
          Safepoint_ops = _vm_queue->drain_at_safepoint_priority ();
          else {safepoint_ops = NULL;
        } while (Safepoint_ops!= NULL);
        _vm_queue->set_drain_list (NULL);
          Safepointsynchronize::end ()//exit security Point} else {//not a safepoint operation if (tracelongcompiles) {
          Elapsedtimer T;
          T.start ();
          Evaluate_operation (_cur_vm_operation);
          T.stop ();
          Double secs = t.seconds (); if (secs * 1e3 > Longcompilethreshold) {TTY->PRINT_CR ("VM%s:%3.7f secs]", _cur_vm_operation->name (
          ), secs);
        } else {evaluate_operation (_cur_vm_operation);
      } _cur_vm_operation = NULL;
 }
    }
    ...
  }

Here is a key explanation for the vm_opration process that is performed within a security point, vmthread to get one or several vm_opertion that need to be executed within a security point by constantly looping, and then preparing to execute these vm_ Before Opration, the Safepointsynchronize::begin () is entered to the safe Point state, and after the vm_opration is executed, the Safepointsynchronize::end () is invoked to exit the safe point mode , and restore all threads that were paused before they continue to run. The logic of this piece of security is pretty complicated, just remember to hold threads_lock this thread mutex when entering the safe point mode, the operation of the thread needs to acquire this lock in order to continue, and also set the state of the security point, if you are entering a safe point in the process of setting _ State is _synchronizing, when all threads have completely entered the safe point to set _state as _synchronized status, when the exit is set to _not_synchronized state.