Druid throws an exception------javax.management.InstanceAlreadyExistsException a series of explorations

The recent project has a scheduled task requirements, timed check MySQL data and ETCD data consistency, specific implementation details will not say, today is to say that the implementation process encountered Druid thrown exception, and the process of solving

Exception 　　 Exception Details

May 05, 2017 4:16:00pm Com.alibaba.druid.proxy.DruidDriver warn warning: Register Druid-driver Mbean Errorjavax.management.InstanceAlreadyExistsException:com.alibaba.druid:type=Druiddriver at Com.sun.jmx.mbeanserver.Repository.addMBean (Repository.java:437) at Com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerWithRepository ( Defaultmbeanserverinterceptor.java:1898) at Com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerDynamicMBean ( Defaultmbeanserverinterceptor.java:966) at Com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerObject (Defaultmbeanserverinterceptor.java :900) at Com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerMBean (Defaultmbeanserverinterceptor.java: 324) at Com.sun.jmx.mbeanserver.JmxMBeanServer.registerMBean (Jmxmbeanserver.java:32t) at Com.alibaba.druid.proxy.DruidDriver.registerDriver (Druiddriver.java:99) at com.alibaba.druid.proxy.druiddriver$1.run (druiddriver.java:84) at java.security.AccessController.doPrivileged (Native Method) at Com.alibaba.druid.proxy.DruidDriver.<clinit> (druiddriver.java:81) at Com.alibaba.druid.pool.DruidDataSource.init (Druiddatasource.java:579) at Com.alibaba.druid.pool.DruidDataSource.getConnection (Druiddatasource.java:986) at Com.alibaba.druid.pool.DruidDataSource.getConnection (Druiddatasource.java:982) at Com.alibaba.druid.pool.DruidDataSource.getConnection (Druiddatasource.java:102) at Org.springframework.jdbc.datasource.DataSourceTransactionManager.doBegin ( Datasourcetransactionmanager.java:211) at Org.springframework.transaction.support.AbstractPlatformTransactionManager.getTransaction ( Abstractplatformtransactionmanager.java:373) at Org.springframework.transaction.interceptor.TransactionAspectSupport.createTransactionIfNecessary ( Transactionaspectsupport.java:447) at Org.springframework.transaction.interceptor.TransactionAspectSupport.invokeWithinTransaction ( Transactionaspectsupport.java:277) at Org.springframework.transaction.interceptor.TransactionInterceptor.invoke (Transactioninterceptor.java: 96) at Org.springframework.aop.framework.ReflectiveMethodInvocation.proceed (Reflectivemethodinvocation.java: 179) at Org.springframework.aop.framework.cglibaopproxy$dynamicadvisedinterceptor.intercept (CglibAopProxy.java: 656) at com.huawei.wgp.task.taskdbetcdconsistencycheck$ $EnhancerBySpringCGLIB $ $e 1d6d636.consistencycheck (<generated>) at Sun.reflect.NativeMethodAccessorImpl.invoke0 (Native Method) at Sun.reflect.NativeMethodAccessorImpl.invoke (Nativemethodaccessorimpl.java:57) at Sun.reflect.DelegatingMethodAccessorImpl.invoke (Delegatingmethodaccessorimpl.java:43) at Java.lang.reflect.Method.invoke (Method.java:30S) at Org.springframework.scheduling.support.ScheduledMethodRunnable.run (Scheduledmethodrunnable.java:65) at Org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run ( Delegatingerrorhandlingrunnable.java:54) at Org.springframework.scheduling.concurrent.ReschedulingRunnable.run (Reschedulingrunnable.java:81) at Java.util.concurrent.executors$runnableadapter.call (Executors.java:471) at Java.util.concurrent.FutureTask.run (Futuretask.java:262) at java.util.concurrent.scheduledthreadpoolexecutor$scheduledfuturetask.access$201 (scheduledthreadpoolexecutor.java:178) at Java.util.concurrent.scheduledthreadpoolexecutor$scheduledfuturetask.run ( Scheduledthreadpoolexecutor.java:292) at Java.util.concurrent.ThreadPoolExecutor.runWorker (Threadpoolexecutor.java:1145) at Java.util.concurrent.threadpoolexecutor$worker.run (Threadpoolexecutor.java:615) at Java.lang.Thread.run (Thread.java:745) May05, 2017 4:16:00Afternoon Com.alibaba.druid.stat.DruidDataSourceStatManager Error severity: Register Mbean Errorjavax.management.InstanceAlreadyExistsException:com.alibaba.druid:type=Druiddatasourcestat at Com.sun.jmx.mbeanserver.Repository.addMBean (Repository.java:437) at Com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerWithRepository ( Defaultmbeanserverinterceptor.java:1898) at Com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerDynamicMBean ( Defaultmbeanserverinterceptor.java:966) at Com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerObject (Defaultmbeanserverinterceptor.java :900) at Com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerMBean (Defaultmbeanserverinterceptor.java: 324) at Com.sun.jmx.mbeanserver.JmxMBeanServer.registerMBean (Jmxmbeanserver.java:522) at Com.alibaba.druid.stat.DruidDataSourceStatManager.addDataSource (Druiddatasourcestatmanager.java:139) at com.alibaba.druid.pool.druiddatasource$1.run (druiddatasource.java:1446) at java.security.AccessController.doPrivileged (Native Method) at Com.alibaba.druid.pool.DruidDataSource.regis Termbean (Druiddatasource.java:1442) at Com.alibaba.druid.pool.DruidDataSource.init (Druiddatasource.java:706) at Com.alibaba.druid.pool.DruidDataSource.getConnection (Druiddatasource.java:986) at Com.alibaba.druid.pool.DruidDataSource.getConnection (Druiddatasource.java:982) at Com.alibaba.druid.pool.DruidDataSource.getConnection (Druiddatasource.java:102) at Org.springframework.jdbc.datasource.DataSourceTransactionManager.doBegin ( Datasourcetransactionmanager.java:211) at Org.springframework.transaction.support.AbstractPlatformTransactionManager.getTransaction ( Abstractplatformtransactionmanager.java:373) at Org.springframework.transaction.interceptor.TransactionAspectSupport.createTransactionIfNecessary ( Transactionaspectsupport.java:447) at Org.springframework.transaction.interceptor.TransactionAspectSupport.invokeWithinTransaction ( Transactionaspectsupport.java:277) at Org.springframework.transaction.interceptor.TransactionInterceptor.invoke (Transactioninterceptor.java: 96) at Org.springframework.aop.framework.ReflectiveMethodInvocation.proceed (Reflectivemethodinvocation.java: 179) at Org.springframework.aop.framework.cglibaopproxy$dynamicadvisedinterceptor.intercept (CglibAopProxy.java: 656) at com.huawei.wgp.task.taskdbetcdconsistencycheck$ $EnhancerBySpringCGLIB $ $e 1d6d636.consistencycheck (<generated>) at Sun.reflect.NativeMethodAccessorImpl.invoke0 (Native Method) at Sun.reflect.NativeMethodAccessorImpl.invoke (Nativemethodaccessorimpl.java:57) at Sun.reflect.DelegatingMethodAccessorImpl.invoke (Delegatingmethodaccessorimpl.java:43) at Java.lang.reflect.Method.invoke (Method.java:30S) at Org.springframework.scheduling.support.ScheduledMethodRunnable.run (Scheduledmethodrunnable.java:65) at Org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run ( Delegatingerrorhandlingrunnable.java:54) at Org.springframework.scheduling.concurrent.ReschedulingRunnable.run (Reschedulingrunnable.java:81) at Java.util.concurrent.executors$runnableadapter.call (Executors.java:471) at Java.util.concurrent.FutureTask.run (Futuretask.java:262) at java.util.concurrent.scheduledthreadpoolexecutor$scheduledfuturetask.access$201 (scheduledthreadpoolexecutor.java:178) at Java.util.concurrent.scheduledthreadpoolexecutor$scheduledfuturetask.run ( Scheduledthreadpoolexecutor.java:292) at Java.util.concurrent.ThreadPoolExecutor.runWorker (Threadpoolexecutor.java:1145) at Java.util.concurrent.threadpoolexecutor$worker.run (Threadpoolexecutor.java:615) at Java.lang.Thread.run (Thread.java:745)

View Code

　　Scene appears

The current development is only a scheduled task, then the project started after the start of the scheduled task, no other operations, and once the scheduled task starts to appear as the exception information, but the scheduled task to complete the normal execution, and does not affect any other service application, and did not throw an exception !

analysis from Druid

Programmers are more or less obsessive-compulsive, and cannot see abnormalities. Although the above exception does not affect the application, but thrown to give me a feeling of discomfort, forcing myself to solve it.

All the anomalies are directed at Druid, so we'll start with Druid. Fortunately, there is a lot of information on the Internet, and finally on Druid's GitHub found javax.management.InstanceAlreadyExistsException anomalies and CentOS release 6.5 always throws Instancealreadyexistsexception These two issue, an article said that the spring automatic detection off, one said the application container in the same instance was launched two times, Caused by the concurrent execution of two scheduled Tasks , the same instance was started two times , this should not be possible, do not explore ( really do not explore it ); then we will close the spring automatic detection to try, As a result, the scheduled task was executed two times. In accordance with other methods provided on the Internet has not been able to solve the problem, I thought this question to help leaders, but God still favor me.

An accidental coincidence, I before the scheduled task, I requested my application from the browser, and then when the scheduled task starts, the exception is not generated, more coincidentally, the data in the database in the browser is repeated, it is said that the scheduled task produced two copies of the same data inserted into the database, Timed tasks have been performed two times! In this sense, the emergence and Druid of the exception is not OK, it is possible that the reason for the scheduled task.

Then the problem shifted, moving from Druid to timed tasks.

analysis from a timed task 　　 1. Do not add timed tasks

<?xml version= "1.0" encoding= "UTF-8"? ><beans xmlns= "Http://www.springframework.org/schema/beans" xmlns: Xsi= "Http://www.w3.org/2001/XMLSchema-instance"Xmlns:context= "Http://www.springframework.org/schema/context" xmlns:task= "Http://www.springframework.org/schema/task"xsi:schemalocation= "Http://www.springframework.org/schema/beans http://www.springframework.org/schema/beans/spring-beans-3.1.xsdhttp//Www.springframework.org/schema/contexthttp//www.springframework.org/schema/context/spring-context-3.1.xsdhttp//Www.springframework.org/schema/taskhttp//www.springframework.org/schema/task/spring-task-3.1.xsd "><!--<bean id= "Taskdbetcdconsistencycheck"class= "Com.yzb.wgp.task.TaskDbEtcdConsistencyCheck"/> <task:scheduled-tasks scheduler= " Schedulerdbetcdconsistencycheck "> <task:scheduled ref=" Taskdbetcdconsistencycheck "method=" ConsistencyCheck "cron=" 0 * * */> </task:scheduled-tasks> <task:scheduler id= "Schedulerdbetcdconsistencycheck" p Ool-size= "2"/>--></beans>

The result is: After startup, no application is dataSource-1, the application is requested, dataSource-1 is initialized only once

May 8:56:12 am Com.alibaba.druid.pool.DruidDataSource info: {dataSource-1} inited

This is a good understanding, the connection pool is lazy loading, the actual use of the time to initialize

　　 2. Timed TasksA), before the scheduled task starts, sends the request to the application Slbadmin

The result is: Initialize dataSource-1 once at the time of the request, then perform a timed task, then initialize the dataSource-1 once, and then perform the scheduled task one at a time.

May 9:21:02 am Com.alibaba.druid.pool.DruidDataSource info: {dataSource-108, 2017 9:22:00 Morning Com.alibaba.druid.pool.DruidDataSource Info: {dataSource-1} inited Scheduled Tasks ...

Here's the question:

1. Why did the scheduled task start 2 times

2, DataSource-1 why will initialize 2 times, supposedly said should only initialize once, this two times is how to appear, feel initialization 3 times than initialize 2 times good understanding (one initialization corresponds to all front-end requests, and 2 initialization for two time tasks respectively)

3, no throw exception: Javax.management.InstanceAlreadyExistsException, this is why

b), do not send any requests to the app Slbadmin before the scheduled task starts

The result is: An exception occurs first, then initializes the dataSource-1 two times, and the last scheduled task executes twice

9:38:00 am Com.alibaba.druid.pool.DruidDataSource info: {dataSource-1-9:38:00  Morning Com.alibaba.druid.pool.DruidDataSource Info: {dataSource-1} inited Scheduled Tasks ... Timed Tasks ...

This is also a question:

1, why there is an exception, and this exception does not terminate the application, the application will be able to provide services normally

2, why is the initialization of dataSource-1 two times after the call two timed tasks, rather than like a to initialize a dataSource-1, call a scheduled task, and then initialize a dataSource-1, and then call a scheduled task

　　 3. Analysis

class= "Com.yzb.wgp.task.TaskDbEtcdConsistencyCheck"/>    <task:scheduled-tasks>        < task:scheduled ref= "Taskdbetcdconsistencycheck" method= "ConsistencyCheck" cron= "0 * *?"/>    </task: scheduled-tasks>    <task:scheduler id= "Schedulerconsistencycheck" pool-size= "1"/>

System.out.println ("thread-id=" + thread.currentthread (). GetId () + "threadname=" + thread.currentthread (). GetName () + ": Timed Task ...");

Print out the thread ID and thread name of the scheduled task, configure and print the code as above, and the result is the same as the 2, plus timed task, just the information we need:

Thread-id=17 threadname=pool-1-thread-1 : Timed Tasks ... Thread-id=18 threadname=pool-2-thread-1: Timed task ...

Thread IDs are different, and the threads pool is different, which means that this is two different threads (the same thread name only spring takes the default naming rules, if we specify the name of the thread in the configuration file, then the thread name is the same), which proves that the same thread is not called two times the target method, So here's the question: how does it generate two threads for timed tasks?

Let's enumerate the resulting scenarios for the possible two threads:

I, configured two scheduled tasks, the target class and the target method is the same, while executing at the same time, like the following configuration

class= "Com.huawei.wgp.task.TaskDbEtcdConsistencyCheck"/>        <task:scheduled-tasks>        < task:scheduled ref= "Taskdbetcdconsistencycheck" method= "ConsistencyCheck" cron= "0 * *?"/>        <task: Scheduled ref= "Taskdbetcdconsistencycheck" method= "ConsistencyCheck" cron= "0 * *?"/>    </task: scheduled-tasks>    <task:scheduler id= "Schedulerconsistencycheck" pool-size= "2"/>

II, there is only one scheduled task in the application, but the application was launched by Tomcat two times

I the scene is easy to exclude, scheduled task configuration file content is very small, very easy to troubleshoot, but how the II scene is generated, then we go to see the Tomcat application list

　　　　 a), non-scheduled Tasks

　　　　 b), before the scheduled task starts, sends the request to the application Slbadmin

　　　 c), do not send the request to the application Slbadmin before the scheduled task starts

Note that the above three pictures of the red place, theoretically speaking, each picture of the second place is not supposed to appear, that is, the path for the Wgp-web line should not exist, then how it appeared? Let's take a look at our project.

This inconsistency is how to produce, the project name and release path is called Wgp-web, the subsequent release path to the Slbadmin, and the project name has not changed, resulting in this inconsistency. Then we'll change the project name to Slbadmin, and then look at the Tomcat app list

The list of applications is normal, and the Slbadmin application is expected to be normal, without throwing an exception, and timed tasks are only executed once at the point.

Questions and Answers 　　 send a request to the app Slbadmin before the scheduled task starts

1. Why did the scheduled task start 2 times

Solution: The same instance in the application container was started two times, resulting in concurrent execution of two scheduled tasks.

2, DataSource-1 why will initialize 2 times, supposedly said should only initialize once, this two times is how to appear, feel initialization 3 times than initialize 2 times good understanding (one initialization corresponds to all front-end requests, and 2 initialization for two time tasks respectively)

Answer: Because the application slbadmin,spring initialized the Slbadmin dataSource-1 before the scheduled task starts, when the Slbadmin timer task starts, it will not initialize Slbadmin dataSource-1 again. However, there are wgp-web scheduled tasks in the container, and when the point is reached, the scheduled tasks of Slbadmin and Wgp-web are started concurrently, because Wgp-web DataSource-1 is not initialized yet. So the timed task executes when initializing the Wgp-web dataSource-1.

3, no throw exception: Javax.management.InstanceAlreadyExistsException, this is why

Answer: dataSource-1 initialization is not concurrent, slbadmin at the time of the request initialization dataSource-1, before the scheduled task execution, and Wgp-web is initialized when the scheduled task starts, Spring will base the two connection pool on the same connection pool based on this time difference. This answer is my guess, a friend who really knows the reason can leave a message!

　 no request is sent to app Slbadmin until the scheduled task starts

1, why there is an exception, and this exception does not terminate the application, the application will be able to provide services normally

Answer: dataSource-1 initialization is concurrent at the same time, this should be said that the spring bug,spring will be implemented in accordance with the Mbean specification beans registered to Mbeanserver, but did not handle the exception, so the tragedy; That is to say Druid is no problem, then the service is able to provide normally.

2, why is the initialization of dataSource-1 two times after the call two timed tasks, rather than like a to initialize a dataSource-1, call a scheduled task, and then initialize a dataSource-1, and then call a scheduled task

Answer: Two timed tasks are performed at the same point, so the initialization of the dataSource-1 is concurrent concurrently, while a) Slbadmin responds to the request before the scheduled task executes and initializes the dataSource-1, Wgp-web is the dataSource-1 that is initialized when a timed task is executed.

Why the previous version did not have this problem: the previous version does not have a scheduled task, in addition, the transfer test version and the live version of the path and display name is consistent, are slbadmin

Summary

　　The root cause of the problem is true: The same instance was started two times,path is/slbadmin boot once, path is/wgp-web boot once,

In the development process, it is best to ensure that the project name and release path ensure that you avoid unnecessary hassles.

Reference

https://github.com/alibaba/druid/issues/1200

https://github.com/alibaba/druid/issues/538

Http://jingyan.baidu.com/article/48206aeaf9422e216ad6b39b.html

https://my.oschina.net/superkangning/blog/467487

Druid throws an exception------javax.management.InstanceAlreadyExistsException a series of explorations