[MySQL] records an online fault handling process-amp; gt; hasmorethan bitsCN.com
[MySQL] records an online fault handling process --> has more than 'max _ user_connections 'active connections
Org. mybatis. spring. MyBatisSystemException: nested exception is org. apache. ibatis. exceptions. PersistenceException:
### Error querying database. cause: org. springframework. transaction. cannotCreateTransactionException: cocould not open JDBC Connection for transaction; nested exception is com. mysql. jdbc. exceptions. jdbc4.MySQLSyntaxErrorException: User NUXUSER already has more than 'max _ user_connections 'active connections
......
### Cause: org. springframework. transaction. cannotCreateTransactionException: cocould not open JDBC Connection for transaction; nested exception is com. mysql. jdbc. exceptions. jdbc4.MySQLSyntaxErrorException: User NUXUSER already has more than 'max _ user_connections 'active connections
......
The above is the email alarm information, and the application page reports the 500,503,504 error message.
1. after receiving the call in the middle of the night, log on to the db immediately and check the status of the db node groups in the 128 groups. we found that db00701 and db10201 had more than 2000 connections.
2 think of the two databases as if they were the db servers that had just maintained ssd last week. directly use ssh to get ahead and check ssh db00701
3. after entering db00701, ps-eaf | grep mysql runs normally. check the error log information and report frequent errors when writing binlog.
4. go to the binlog directory/data/binlog/and find that there are more than 100 binlogs under the Directory, which is more than five days long. how can this happen, didn't a crontab task remove the binlog two days ago every day? Immediately crontab-l found that the task for removing the binlog was not enabled. suddenly, I thought that the disk of the binlog directory would not be full, right?
5. run df-h and check that/data/binlog/is indeed used 100%. then, manually remove the binlog. After 3 minutes, the application will be normal and the 5XX error will not be reported on the page, db00701 and db10201 connections are also normal.
There are two reasons to summarize the problem:
1. during this database db server switch, Uncle Yamu's sa group did not add the binlog monitoring feature to the monitoring feature, so the disk was not promptly notified of alarms.
2. our db Group does not have the double confirmed monitor in time.
BitsCN.com