A case of Oracle database exception restart caused by VCs

1. Environmental description

Operating system version: SUSE Linux Enterprise Server SP2 (x86_64)

Database version: Oracle 11.1.0.7.16

VCs version: 5.1

Blog Address: http://blog.csdn.net/hw_libo/article/details/41171561

2. Problem phenomena and analysis

About 3:46, the SMS received the database abnormal alarm.

(1) View Database alert log

Sun Nov 03:46:51 2014Stopping Background process smcostopping background process fbdashutting down Instance:further Lo Gons Disabledsun Nov 03:46:54 2014kkjcre1p:unable to spawn jobq slave process, slot 0, error 1089Stopping background p rocess cjq0stopping Background Process qmncstopping Background process mmnlstopping background process Mmonsun Nov 16 03:4 7:03 2014Shutting Instance (immediate) License high water mark = 366Sun Nov 03:51:52 2014License High water mark = 366USER (ospid:8654): Terminating the Instancesun Nov 03:51:52 2014ORA-1092:OPIODR aborting process unknown ospid (2 8656_47367034883680) Sun Nov 03:51:52 2014LGWR waiting for instance termination

No obvious cause of abnormality was found.

We also looked at the report for that time period and found that the database was under a little pressure and there was nothing unusual about SQL.

(2) Check OS Resource Usage

System resource usage is normal, IO is not high.

(3) View messages log

Nov 03:42:46 Boscodb agentframework[10894]: VCS ERROR v-16-2-13027 Thread (4149214112) Resource (ORA_VG)-Monitor proced Ure did not complete within the expected time.  Nov 03:42:46 Boscodb had[10258]: VCS ERROR v-16-2-13027 (boscodb) Resource (ORA_VG)-monitor procedure did not complete Within the expected time. Nov 03:43:01 boscodb su: (to root) root on none ... Nov 03:45:39 boscodb su: (to Oracle) root on Nonenov 03:45:47 boscodb agentframework[10894]: VCS ERROR v-16-2-13210 Thread (4154813344) Agent is calling clean for resource (ORA_VG) because 4 successive invocations of the monitor procedure D ID not complete within the expected time.  Nov 03:45:47 Boscodb had[10258]: VCS ERROR v-16-2-13210 (BOSCODB) Agent is calling clean for resource (ORA_VG) because 4 Successive invocations of the monitor procedure did not complete within the expected time .... Nov 03:46:48 Boscodb agentframework[10894]: VCS ERROR v-16-2-13006 Thread (4153760672) Resource (ORA_VG): Clean Procedur E did not cOmplete within the expected time. Nov 03:46:48 Boscodb had[10258]: VCS ERROR v-16-2-13006 (boscodb) Resource (ORA_VG): Clean procedure didn't complete WI Thin the expected time. Nov 03:46:49 Boscodb agentframework[10899]: VCS ERROR v-16-2-13067 Thread (4155194272) Agent is calling clean for resour CE (ora_archive) because the resource became OFFLINE unexpectedly, on its own. Nov 03:46:50 Boscodb had[10258]: VCS ERROR v-16-2-13067 (BOSCODB) Agent is calling clean for resource (ora_archive) Beca Use the resource became OFFLINE unexpectedly and on its own. Nov 03:46:51 Boscodb agentframework[10899]: VCS ERROR v-16-2-13068 Thread (4155194272) Resource (ora_archive)-Clean com Pleted successfully.

It turned out that the VCs had an error, the monitor VG timed out, and these error messages are also recorded in the VCs log.

3. Workaround

Check the moniter script to monitor the VG, found that there is no exception, and the script in other libraries, the pressure is relatively large has not been a problem.

The current configuration is:

        Application ORA_VG (                startprogram = "/opt/bosc O_o ra/bin/vg_start.sh"                stopprogram = "/opt/bosc O_o ra/bin/vg_ stop.sh "                cleanprogram ="/opt/bosc O_o ra/bin/vg_clean.sh "                monitorprogram ="/opt/bosc O_o ra/bin/vg_ monitor.sh "                monitorinterval =                monitortimeout = +                )

The decision to temporarily increase the monitoring duration to 90s can be used in two ways:

(1) can modify MAIN.CF, but need to restart VCs, no Operation time window

(2) Dynamic modification of the command

Haconf-makerw

Hares-modify ORA_VG monitortimeout 90 # # Adjustment monitoring length of 90s, default is 60s

Haconf-dump

Blog Address: http://blog.csdn.net/hw_libo/article/details/41171561

--Bosco qq:375612082

----END----
-------------------------------------------------------------------------------------------------------
Copyright, the article allows reprint, but must be linked to the source address, otherwise investigate legal responsibility!

A case of Oracle database exception restart caused by VCs