An example of abnormal restart of the oracle database caused by VCS: vcsoracle

An example of abnormal restart of the oracle database caused by VCS: vcsoracle
1. Environment Description

Operating system version: SUSE Linux Enterprise Server 10 sp2 (x86_64)

Database Version: Oracle 11.1.0.7.16

VCS version: 5.1

Blog: http://blog.csdn.net/hw_libo/article/details/41171561

2. Problem phenomena and Analysis

At about a.m., the SMS received an exception alert from the database.

(1) view Database alert logs

Sun Nov 16 03:46:51 2014Stopping background process SMCOStopping background process FBDAShutting down instance: further logons disabledSun Nov 16 03:46:54 2014kkjcre1p: unable to spawn jobq slave process, slot 0, error 1089Stopping background process CJQ0Stopping background process QMNCStopping background process MMNLStopping background process MMONSun Nov 16 03:47:03 2014Shutting down instance (immediate)License high water mark = 366Sun Nov 16 03:51:52 2014License high water mark = 366USER (ospid: 8654): terminating the instanceSun Nov 16 03:51:52 2014ORA-1092 : opiodr aborting process unknown ospid (28656_47367034883680)Sun Nov 16 03:51:52 2014LGWR waiting for instance termination

No obvious causes of exceptions were found.

At the same time, I also checked the reports for that time period and found that the database was under little pressure and there was no abnormal SQL.

(2) Check OS resource usage

System resources are normally used, and IO is not high.

(3) view the messages log

Nov 16 03:42:46 boscodb AgentFramework[10894]: VCS ERROR V-16-2-13027 Thread(4149214112) Resource(ora_vg) - monitor procedure did not complete within the expected time.Nov 16 03:42:46 boscodb Had[10258]: VCS ERROR V-16-2-13027 (boscodb) Resource(ora_vg) - monitor procedure did not complete within the expected time.Nov 16 03:43:01 boscodb su: (to root) root on none……Nov 16 03:45:39 boscodb su: (to oracle) root on noneNov 16 03:45:47 boscodb AgentFramework[10894]: VCS ERROR V-16-2-13210 Thread(4154813344) Agent is calling clean for resource(ora_vg) because 4 successive invocations of the monitor procedure did not complete within the expected time.Nov 16 03:45:47 boscodb Had[10258]: VCS ERROR V-16-2-13210 (boscodb) Agent is calling clean for resource(ora_vg) because 4 successive invocations of the monitor procedure did not complete within the expected time.……Nov 16 03:46:48 boscodb AgentFramework[10894]: VCS ERROR V-16-2-13006 Thread(4153760672) Resource(ora_vg): clean procedure did not complete within the expected time.Nov 16 03:46:48 boscodb Had[10258]: VCS ERROR V-16-2-13006 (boscodb) Resource(ora_vg): clean procedure did not complete within the expected time.Nov 16 03:46:49 boscodb AgentFramework[10899]: VCS ERROR V-16-2-13067 Thread(4155194272) Agent is calling clean for resource(ora_archive) because the resource became OFFLINE unexpectedly, on its own.Nov 16 03:46:50 boscodb Had[10258]: VCS ERROR V-16-2-13067 (boscodb) Agent is calling clean for resource(ora_archive) because the resource became OFFLINE unexpectedly, on its own.Nov 16 03:46:51 boscodb AgentFramework[10899]: VCS ERROR V-16-2-13068 Thread(4155194272) Resource(ora_archive) - clean completed successfully.

It turns out that VCS has encountered an error and the monitoring vg has timed out. These error messages are also logged in the VCS logs.

3. Solution

I checked the moniter script that monitors vg and found that there were no exceptions. In addition, there were no problems in the pressure of the script in other libraries.

The current configuration is:

        Application ora_vg (                StartProgram = "/opt/BOSCO_ora/bin/vg_start.sh"                StopProgram = "/opt/BOSCO_ora/bin/vg_stop.sh"                CleanProgram = "/opt/BOSCO_ora/bin/vg_clean.sh"                MonitorProgram = "/opt/BOSCO_ora/bin/vg_monitor.sh"                MonitorInterval = 30                MonitorTimeout = 30                )

You can use either of the following methods to increase the monitoring duration to 90 s:

(1) You can modify main. cf, but you need to restart VCS without an operation time window.

(2) dynamically modify commands

Haconf-makerw

Hares-modify ora_vg MonitorTimeout 90 # adjust the monitoring duration to 90 s. The default value is 60 s.

Haconf-dump

Blog: http://blog.csdn.net/hw_libo/article/details/41171561

-- Bosco QQ: 375612082

---- END ----
Bytes -------------------------------------------------------------------------------------------------------
All rights reserved. reprinted articles are allowed, but source addresses must be indicated by links. Otherwise, the documents will be held legally responsible!