The theory and practice of Java thread pool

Some time ago, there was a project in the company that needed to be refactored, with the goal of improving throughput and usability, modifying the original threading model and processing logic in this process, and discovering that there are many basics of multithreading that are blurred, such as the operation of the underlying thread, the policies and logics of the existing thread pool, The monitoring of the health status of the thread in the pool, and so on, this time again reviewed, which involves a large number of classes in the Java.util.concurrent package. This article will contain the following content:
The relationship between thread in Java and threads in the operating system
Various overhead for thread switching
The significance of the existence of Threadgroup
Using the thread pool to reduce threading overhead
The concept of executor
Some concrete implementations in the Threadpoolexecutor
How to monitor the health of threads
Refer to Threadpoolexecutor to design a threading model that suits you
First, the problem description

The software architecture of the project's system (from development to operation) is basically a microservices architecture, which solves the complexities of our system, but it also poses some problems, such as that most of the services in this architecture have their own separate databases, while some (very important) businesses need to do cross-library queries. Believe that this "cross-Library query" problem Many practice microservices companies have encountered, usually this kind of problem has the following solutions (of course, there are many other solutions, here is not described here):

API queries are strictly provided through the service.
The advantage of this is that the service is fully treated as a black box, minimizing dependency and coupling between services, and then using different database types depending on the actual needs of the service, and the disadvantage is that it is too expensive.

Redundancy of the information you care about into your own library, and provide APIs for other services to proactively modify.
The advantage is that information updates are very real-time, with the disadvantage of increasing dependency between services.

Command and query separation (CQRS). Data warehouses that may be cared for by other services are put into the data warehouse (or made similar to materialized views, search engines, etc.), and the Data Warehouse provides only read functionality.
The advantage is that there will be no pressure on the main library, service as long as the care to realize their own business is good, the disadvantage is that the real-time data will be challenged.

Command and query separation
In combination with the actual situation, we are using the 3rd option. However, as more and more businesses rely on reading the library, and even rely on some of the changes in the state, so the data synchronization of the reading library if there is a high latency, it will directly affect the conduct of the business. After several such events, he was determined to improve the situation. The first thing to think about is the consumption of messages using line pool (written to read library), which has provided a practical and powerful thread pool tool--executor framework since 1.5.

Second, executor framework

The executor framework was introduced in Java1.5, and most of the classes are in Package Java.util.concurrent, written by the great God Doug Lea, which is commonly used to have the following classes and interfaces:

Java.util.concurrent.Executor
An interface that contains only one method, the abstract meaning of which is: the executor used to perform a runnable task.

Java.util.concurrent.ExecutorService
An extension to executor adds many interfaces for managing the life cycle of tasks and actuators, and is often the most commonly used interface for multithreaded development.

Java.util.concurrent.ThreadFactory
An interface that generates a new thread. The user can manage the logic of generating threads in the thread pool by implementing this interface

Java.util.concurrent.Executors
Provides a number of practical methods for generating actuators, such as the implementation of a thread-pool-based executor.

Third, why use thread pool

Java is thread-based from the very beginning, and threads are encapsulated in Java as a class java.lang.Thread. In the interview, many interviewers will ask a very basic question about threading:

There are several ways to create a new thread in Java?
As everyone knows, the standard answer is two types: inheriting thread or implementing Runnable, which is also written in the thread class comment in the JDK source code.

However, in my opinion, these two methods are basically a kind, all want to open the operation of the thread, must produce a thread class (or its subclass) instance, executes the native method Start0 ().

Threads in Java

Java in the abstraction of the thread into a common class, which brings a lot of benefits, such as a simple use of object-oriented methods to achieve multi-threaded programming, but this program is much easier to forget that the object at the bottom of the actual corresponding to a thread in the OS.

Threads and processes in the operating system

Process can be seen as a JVM, and it can be seen that all processes have their own private memory, which has a mapping in main memory, and all threads share memory in the JVM. In modern operating systems, the scheduling of threads is usually integrated into the operating system, and the operating system can analyze more information to determine how to schedule threads more efficiently, which is why Java insists that the order of execution of threads is not guaranteed, because the JVM cannot control this itself. So you can only think of it as completely disordered.

In addition, many of the properties in class Java.lang.Thread are mapped directly to some of the properties of the threads in the operating system. Some of the methods provided in Java thread, such as sleep and yield, depend on the scheduling algorithm of the thread in the operating system.

About the thread scheduling algorithm can read the operating system related books, here do not do too much narration.

The cost of the thread

Typically, the context switch between threads in the operating system consumes about 1 to 10 microseconds
From there, you can see that the thread contains some contextual information:

CPU stack pointer (stack),
The value of a set of registers (registers),
The value of the instruction counter (PC), etc.,
They are all stored in the main memory of the process in which the thread resides, and for Java, this process is the process where the JVM resides, and the runtime memory of the JVM can be easily divided into the following sections:

Several stacks (stack). Each thread has its own stack, the stack in the JVM cannot store objects, and only the underlying variables and object references can be stored.
Heap. A JVM has only one heap, and all objects are allocated on the heap.
Method area. A JVM has only one method area, which contains the bytecode and static variables for all loaded classes.
The stack in # # can be thought of as the context of this thread, the creation thread to request the corresponding stack space, and the size of the stack space is certain, so when the stack space is not enough, it will cause the thread request is unsuccessful. As you can see in the source code of thread, the last step in starting a thread is to execute a local method private native void Start0 (), and code 1 is the method that start0 the final call in OpenJDK:

Code 1
Jvm_entry (void, jvm_startthread (jnienv env, Jobject jthread))
Jvmwrapper ("Jvm_startthread");
Javathread native_thread = NULL;
BOOL Throw_illegal_thread_state = false;

We must release the Threads_lock before we can post a JVMTI event
In Thread::start.
{
Mutexlocker Mu (threads_lock);

//省略一些代码  jlong size =         java_lang_Thread::stackSize(JNIHandles::resolve_non_null(jthread));  size_t sz = size > 0 ? (size_t) size : 0;  native_thread = new JavaThread(&thread_entry, sz);

}

if (native_thread->osthread () = = NULL) {
Throw_msg (Vmsymbols::java_lang_outofmemoryerror (),
"Unable to create new native thread");
}

Thread::start (Native_thread);

Jvm_end
As you can see from code 1, the creation of threads requires stack space first, so excessive thread creation can cause oom.

At the same time, thread switching has the following overhead:

The switchover of the execution context in the CPU causes the interrupt of the "instruction pipelining (instruction Pipeline)" in the CPU and the failure of the CPU cache.
If there are too many threads, thread switching takes longer than the thread executes, which is a serious waste of CPU resources.
Competition for shared resources (locks) can cause a sharp increase in thread switching overhead.
Based on the above description, it is generally recommended to create as few threads as possible, reduce the use of locks (especially synchronized), and use the synchronization tools provided by the JDK as much as you can. In order to reduce the overhead of thread context switching, it is often a valid method to use the thread pool.
With 1-5 working experience, in the face of the current popular technology do not know where to start, need to break through the technical bottleneck can add group. Stay in the company for a long time, have a very comfortable, but job-hopping interview wall. Need to study in a short period of time, job-hopping to get a high salary can add group. If there is no work experience, but the foundation is very solid, on the Java work mechanism, common design ideas, Common Java Development Framework Master skilled can add group. Java Architecture Group: 5,825,056,431 communication.

The thread pool in Java

The most common in the executor framework is probably java.util.concurrent.ThreadPoolExecutor, and for its description, it simply says, "It maintains a thread pool, for tasks submitted to this executor, Instead of creating a new thread, it executes using a thread in the pool. For tasks with a "large number but little execution time", you can significantly reduce the overhead of task execution. Java.util.concurrent.ThreadPoolExecutor contains a number of attributes that developers can use to customize different thread pool behavior, as follows:

1. Size of thread pool: corepoolsize and Maximumpoolsize

The size of the threadpoolexecutor thread pool is determined by these two properties, which refers to the minimum (core) number of threads when a thread pool is functioning properly, and when a task arrives, a new thread is generated if the current pool has fewer threads than corepoolsize The latter refers to the maximum number of threads that can be generated when the waiting queue is full. The two values are equal in the object returned in Example 1, equal to the value passed in by the user.

2. The user can launch the core thread by invoking an instance method on Java.util.concurrent.ThreadPoolExecutor

3. Customizable thread Generation: threadfactory

The default thread is created by the Threadfactory returned by Method Executors.defaultthreadfactory (), and the thread created by default is not daemon, and the developer can pass in a custom threadfactory to customize the thread.

5. Idle wait time for non-core threads: KeepAliveTime

6. Task Wait queue: WorkQueue

This queue is an instance of java.util.concurrent.blockingqueue<e>. When there are currently no idle threads in the pool to perform tasks, the task is put into a wait queue, which can be divided into 3 different queue policies depending on the implementation class:

Capacity is 0. such as: Java.util.concurrent.SynchronousQueue
Waiting for a queue capacity of 0, all tasks that need to be blocked must wait for a thread in the pool to be idle before proceeding or blocking. The thread pool generated by calling Executors.newcachedthreadpool's two function is this policy.

Unlimited capacity. Example: Java.util.concurrent.LinkedBlockingQueue with no specified capacity
The length of the wait queue is infinitely large, according to the above-mentioned strategy, there will be no more than corepoolsize threads are created, so maximumpoolsize is meaningless. Calling the thread pool generated by Executors.newfixedthreadpool is this policy.

Limit capacity. such as: Any java.util.concurrent.blockingqueue<e> of the specified capacity
In some scenarios (described in this article), you need to specify the capacity of the wait queue to prevent excessive resource consumption, such as the use of an unlimited waiting queue, when there are a large number of tasks coming and there are no idle threads in the pool to perform tasks, there will be a lot of task accumulation, which are objects of a class. is to consume memory, it can cause oom. How to balance the waiting queue and the size of the thread pool depends on the actual scenario, and if configured improperly, it can result in resource exhaustion, thread context switching consumption, or thread scheduling consumption. These will directly affect the throughput of the system.

7. Task Reject Processor: DefaultHandler

If the task is denied execution, the Rejectedexecutionhandler.rejectedexecution () method on the object is called, and the JDK defines 4 processing policies that the user can customize for their own task-handling policies.

8. Allow core threads to expire: allowcorethreadtimeout

All of the above is based on this variable is false (the default), if your thread pool is no longer used (not referenced), but there are still alive threads, the thread pool is not recycled, this situation caused a memory leak-- A piece of memory that is never accessed will not be recycled by GC.
The user can either release it explicitly by calling shutdown () when the thread pool is discarded, or set allowcorethreadtimeout to true, and the core thread will be freed after the expiration time, and it will be recycled by GC.

Iv. What to do if the thread dies

In almost all comments on the method that generates the thread pool in the executors, there is a sentence that represents the same meaning, indicating that if a thread in the thread pool dies, it generates a new thread instead. The following is a comment on method java.util.concurrent.Executors.newFixedThreadPool (int).

If any thread terminates due to a failure during execution prior to shutdown, a new one would take it place if needed to E Xecute subsequent tasks.
Causes of thread death

We all know that the daemon thread (daemon) will die after all the non-daemon threads have died, in addition to the following possible reasons for a non-daemon thread to die:

Natural death, the Runnable.run () method is returned after execution is complete.
There was an uncaught exception during execution and was thrown out of Runnable.run (), causing the thread to die.
The host dies, the process shuts down, or the machine freezes. In Java, usually the System.exit () method is called
Other hardware issues.
To ensure high availability of the thread pool, it is necessary to ensure that the thread is available. Like a fixed-capacity thread pool, where one thread dies, it must be able to monitor the death of the thread and generate a new thread to replace it. There are several concepts related to threading in Threadpoolexecutor:

Java.util.concurrent.ThreadFactory, there are two types of threadfactory in executors, but the thread pool provided uses only one Java.util.concurrent.Executors.DefaultThread Factory, it is simple to use threadgroup to achieve.

Java.lang.ThreadGroup, a class that exists from the beginning of JAVA1, is used to create a tree of threads that can be used to organize relationships between threads, but it does not monitor the child threads it contains.

Java.util.concurrent.threadpoolexecutor.worker,threadpoolexecutor encapsulates the thread, which also contains some statistical functions.

How to guarantee the availability of threads in Threadpoolexecutor

In Threadpoolexecutor, a clever way to monitor the thread health of threads pool is implemented, code 2 is a piece of code that is intercepted from the Threadpoolexecutor class source, which together illustrates its monitoring of threads.

As you can see, the threads in Threadpoolexecutor are encapsulated into an object worker, and the run () agent in the Threadpoolexecutor is Runworker () in the Runworker () Method is a dead loop that gets the task and executes it. If a task is running out of the question (such as throwing an uncaught exception), the Processworkerexit () method is executed, and the incoming completedabruptly parameter is true, and a worker with the initial task null is re-added. And start a new thread with it.

Code 2
Dynamic inner class of Threadpoolexecutor
Private Final class Worker extends Abstractqueuedsynchronizer implements Runnable {

/** 对象中封装的线程 */final Thread thread;/** 第一个要运行的任务，可能为null. */Runnable firstTask;/** 任务计数器 */volatile long completedTasks;//省略其他代码Worker(Runnable firstTask) {    setState(-1); // inhibit interrupts until runWorker    this.firstTask = firstTask;    this.thread = getThreadFactory().newThread(this);}/** Delegates main run loop to outer runWorker  */public void run() {    runWorker(this);}

}

final void Runworker (Worker w) {
Thread wt = Thread.CurrentThread ();
Boolean completedabruptly = true;
try {
while (task! = NULL | | (Task = Gettask ()) = null) {
W.lock ();
try {
BeforeExecute (WT, Task);
try {
Task.run ();
} finally {
AfterExecute (task, thrown);
}
} finally {
task = null;
w.completedtasks++;
W.unlock ();
}
}
completedabruptly = false;
} finally {
Processworkerexit (w, completedabruptly);
}
}

private void Processworkerexit (Worker W, Boolean completedabruptly) {
if (Runstatelessthan (c, STOP)) {
if (!completedabruptly) {
int min = allowcorethreadtimeout? 0:corepoolsize;
if (min = = 0 &&! workqueue.isempty ())
min = 1;
if (Workercountof (c) >= min)
Return Replacement not needed
}
Addworker (null, FALSE);
}
}

Private Boolean Addworker (Runnable Firsttask, Boolean core) {
Retry:
for (;;) {
int c = Ctl.get ();
Int rs = runstateof (c);

    Check If queue empty only if necessary. if (rs >= SHUTDOWN &&!        (rs = = SHUTDOWN && Firsttask = = null &&! workqueue.isempty ()))    return false; for (;;)        {int WC = Workercountof (c);            if (WC >= Capacity | |        WC >= (Core corepoolsize:maximumpoolsize)) return false;        if (Compareandincrementworkercount (c)) break retry;  c = Ctl.get ();        Re-read CTL if (runstateof (c)! = RS) continue retry; Else CAS failed due to workercount change; Retry Inner loop}}boolean workerstarted = False;boolean workeradded = false;    Worker W = null;try {w = new Worker (firsttask);    Final Thread t = w.thread;        if (t! = null) {final Reentrantlock mainlock = This.mainlock;        Mainlock.lock ();            try {//recheck while holding lock. Threadfactory failure or if//Shut down before LoCK acquired.            int rs = runstateof (Ctl.get ());                if (Rs < SHUTDOWN | | (rs = = SHUTDOWN && Firsttask = = null)) {if (t.isalive ())//PreCheck that T is startable throw new illegalthreadstateexception                ();                Workers.add (w);                int s = workers.size ();                if (S > largestpoolsize) largestpoolsize = s;            Workeradded = true;        }} finally {Mainlock.unlock ();            } if (workeradded) {T.start ();        Workerstarted = true; }}} finally {if (! workerstarted) addworkerfailed (w);} return workerstarted;

}
Five, back to my question

For a variety of reasons, we did not use the database's own master-slave mechanism to replicate the data, but instead, all the DML statements of the main library are sent as messages to the Read library (DTS), and the replay of the data is realized. The first version of the data Synchronization service is simple, DML message processing and consumption (written to read libraries) for the main library are done within one thread. The advantage of this implementation is simple, but the disadvantage is directly caused by the table and the data synchronization between the table will be affected, if a table a suddenly come a lot of messages (often caused by bulk modification of data), will occupy the message processing channel, affecting other business data and While the single-thread write-through is too small.

As mentioned above, the first thought is to use the thread pool to do the message consumption, but can not directly apply the above-mentioned executor framework, for the following reasons:

Threadpoolexecutor all the tasks in the default are not mutually affected, but for the database DML, the order of the messages cannot be disrupted, at least the order of the messages in a single table must be ordered, otherwise it will affect the final data consistency.
All threads in the Threadpoolexecutor share a wait queue, but each thread should have its own task waiting queue in order to prevent the table from being affected by the table.
The throughput of the write library operation is directly affected by the number of committed transactions, so this multithreaded framework can support the merging of tasks.
It makes no sense to repeat the wheel, but in our scenario, the existing executor framework in the JDK does not meet the requirements and can only build wheels on its own.

With 1-5 working experience, in the face of the current popular technology do not know where to start, need to break through the technical bottleneck can add group. Stay in the company for a long time, have a very comfortable, but job-hopping interview wall. Need to study in a short period of time, job-hopping to get a high salary can add group. If there is no work experience, but the foundation is very solid, on the Java work mechanism, common design ideas, Common Java Development Framework Master skilled can add group. Java Architecture Group: 5,825,056,431 communication.
My implementation

First, the thread is abstracted into an executor (Executor) of the "DML statement. It contains an instance of thread, maintains its own waiting queue (blocking queue with capacity limit), and corresponding message execution logic.

In addition, there are some simple statistics, thread health monitoring, merging transactions and other processing.

The Executor object implements the Thread.uncaughtexceptionhandler interface and binds it to its worker thread. The Executorgroup also regenerates into a daemon thread dedicated to guarding all threads within the pool as an additional insurance measure.
The concept of the thread pool is abstracted into an actuator group (Executorgroup), which maintains an array of actuators, maintains the mapping of the target table to a particular executor, and provides an interface for executing messages externally, with the main code as follows:

Code 3
public class Executorgroup {

Executor[] group = new Executor[NUM];Thread boss = null;Map<String, Integer> registeredTables = new HashMap<>(32);

Atomicinteger cursor = new Atomicinteger ();
volatile int cursor = 0;

Public Executorgroup (String name) {//init Group for (int i = 0; i < NUM; i++) {logger.debug ("boot thread {},{}",        name, i);    Group[i] = new Executor (This, String.Format ("sync-executor-%s-%d", Name, I), i/num_of_first_class); } Startdaemonboss (String.Format ("Sync-executor-%s-boss", Name));}    Extra Insurance private void Startdaemonboss (String name) {if (boss! = null) {boss.interrupt ();            } boss = new Thread ((), {while (true) {//Rest one minute ... if (this.group! = null) {for (int i = 0; i < group.length; i++) {Executor Executor                    = Group[i];                    if (executor! = null) {Executor.checkthread ();    }                }            }        }    });    Boss.setname (name);    Boss.setdaemon (TRUE); Boss.start ();}    public void execute (Message message) {Logger.debug ("execute message"); Omitting message legality validation if (!registeredtables.containskey (Taskkey)) {//Registered

Registeredtables.put (Taskkey, Cursor.getandincrement ());
Registeredtables.put (Taskkey, cursor++% NUM);
}
int index = Registeredtables.get (Taskkey);
Logger.debug ("execute message {}, register index {}", Taskkey, index);
try {
Group[index].schedule (message);
} catch (Interruptedexception e) {
Logger.error ("Error Preparing message", e);
}

}

}
After completion, the overall threading model is as follows:

The new threading model
Java1.7 newly added Transferqueue

The new queue type Transferqueue is provided in Java1.7, but only provides a java.util.concurrent.linkedtransferqueue<e> of its implementation, and it has better performance, But it is a non-capacity queue, and in our scenario we have to limit the capacity of the queue, so we have to implement a capacity-constrained queue by ourselves.

The theory and practice of Java thread pool