Problems with Java thread Pool resource reclamation

Recent projects in order to improve the user experience, the foreground to create task background tasks, with multi-threading to run.

now the scene : Background timer task management These two thread pools, a maximum number of threads 10, a maximum number of threads 15. After the application is deployed, it is not more than 5 hours, server load is high, memory usage is excessive.

Analysis Reason : Because this feature is the Excel import function, if the foreground has a large number of import tasks, then the background load will be very high.

　　　　　My implementation principle : Timed task read tasks, put in the Task Queue table, and then use the thread pool to consume tasks in the task queue, each thread has been iterating over the task.

My scheduled task cycle is 1 minutes, when the thread pool is initialized, the idle thread survives for 1 minutes, and the task queue does not set a maximum value.

optimization :

　　1. No upper limit for task queue

Each time the scheduled task goes back to the database to read the task, put into the list, if the task queue, then the task queue occupies more and more memory, resulting in insufficient server memory.

　　　workaround : When the task is read, the task queue size, such as the 100 I set, then I will not read the new task. This parameter needs to be adjusted by observing the load of the machine.

　　2. Thread in thread pool , infinite loop Processing task, when the number of tasks is too high, the thread will execute and cannot stop.

Thread in thread pool, infinite loop processing task, when the number of tasks is too high, the thread will be executed and cannot be stopped.

　　　workaround : Set the counter internally, when a thread accumulates to a certain number, exits the loop, and then empties the resources, recycles, intermittently, and periodically performs tasks that are equivalent to regular recycling of resources

　　3. Scheduled Tasks 1 minutes, then the idle thread survives for 1 minutes

Since the idle thread survives for 1 minutes, then my scheduled task is 1 minutes, and at this point the thread will not be recycled by the thread pool, so the resource is not released and recycled.

　　　workaround : Adjust thread idle threads to survive for half of the scheduled task cycle, that is, 30 seconds.

　　4. In the thread pool, there are core threads that are also recycled for core blocks until those, so the following method needs to be performed to ensure that the core line is blocks until those and then recycled.

　　　workaround : Threadpoolexecutor.allowcorethreadtimeout (True);

the process after optimization :

The task periodically reads the task, puts a certain amount (not exceeding the maximum) in the task queue, and then the thread in the pool blocks until those a shorter time, the thread passes through the counter, and when a certain number of threads are executed, the thread is idle, the back is reclaimed by the thread pool, the resource is recycled, So repeatedly, resources are recycled, redistributed, and will not consume much of the service resources.

Through the above several points of optimization, the program consumes a lot of server resources, there may be a place to optimize, and then add.

Problems with Java thread Pool resource reclamation