Scenario:
After the program is modified, upload it to tomcat on the application server. After restarting tomcat, an exception occurs! The front-end nginx Server Load balancer has a 504 error and the website cannot be opened!
Troubleshooting:
After troubleshooting tomcat logs, it is found that the database is connected to an exception. After logging on to the database, the "conn/as sysdba" method is used to log on normally, and the database is light loaded without pressure. Therefore, the listener of the database is checked, run the "lsntctl services" command to get stuck. The result is unavailable for half a day...
Troubleshooting:
Enter the "lsntctl stop" or "lsntctl reload" command to get stuck. If the result does not appear for a long time, the system will not be able to get the result! Enter the "netstat-ntpl | grep: 1521" command to find the process Number of the listener. kill-9 stops, restart the listener, and then start tomcat. The website is accessible!
Cause analysis:
The database connection may fail because the listener is stuck due to a database bug! See:
Post-maintenance:
To avoid this problem as much as possible, I want to write a monitoring script in combination with nagios monitoring. When the database connection is abnormal, I can receive an alert text message as soon as possible. java programs connect to the database and use the connection pool, therefore, it may not be possible to expose problems in time!
Write a script and run it in the task plan. regularly connect to the database, query the system time and database status, and output the spool to the temporary file.
- [Root @ server199 ~] # Crontab-l
- */5 */usr/local/nagios/libexec/connect_Oracle.sh
- [Root @ server199 ~] # Cat/usr/local/nagios/libexec/connect_oracle.sh
#! /Bin/sh
# Functions: connect oracle server test
# Author: lw. yang
# Modify_date: 2012-02-22
- Rm-rf/tmp/check_oracle.log
Export ORACLE_HOME =/u01/app/oracle/product/10.2.0/db_1/
$ ORACLE_HOME/bin/sqlplus username/password @ ip: 1521/services_name <EOF
Set echo off
Set feedback off
Spool/tmp/check_oracle.log
Alter session set nls_date_format = 'yyyy-MM-DD: HH24: MI: ss ';
Select sysdate from dual;
Select name, open_mode from v \ $ database;
Spool off
Set echo on
Set feedback on
EOF