Shutdown immediate when hang (reprint)

Shutdown immediate
Hang at regular Guanqu, in alert

License High Water mark = 4
All dispatchers and shared servers shutdown

Wait a little more, you'll be there.
Shutdown:active processes Prevent SHUTDOWN operation

The cause of this phenomenon is (and may be, the cause of EM, which is not related to EM):

The previous session was not disconnected and then switched to the OS prompt using host, which prevented the database from shutting down properly

[Email protected] ~]$ Sqlplus Scott/tiger

Sql> Show User
USER is "SCOTT"
Sql>!
[Email protected] ~]$ echo $$
3897
[Email protected] ~]$ Sqlplus/as SYSDBA

Sql> Select Pid,spid,p.pname,p.program from v$process p,v$session s where p.addr=s.paddr and s.sid=userenv (' Sid ');

PID SPID PNAME Program
---------- ------------------------ ----- ------------------------------------------------
3911 [email protected] (TNS v1-v3)

sql> shutdown Immediate

Hang on

Alert content:

SAT Jan 04 17:41:53 2014
Shutting down instance (immediate)
Stopping background process SMCO
Shutting down Instance:further logons disabled
SAT Jan 04 17:41:53 2014
Stopping background process CJQ0
Stopping background process QMNC
Stopping background process MMNL
Stopping background process Mmon
License High Water mark = 4
All dispatchers and shared servers shutdown

SAT Jan 04 17:46:58 2014
Shutdown:active processes Prevent SHUTDOWN operation

SAT Jan 04 17:52:00 2014
Shutdown:active processes Prevent SHUTDOWN operation

Another terminal
$ pstree-p

├─sshd (1668) ─┬─sshd (1838) ───sshd (1840) ─┬─bash (3067) ───tail (3095)
│            │                         └─bash (3106) ───rlwrap (3893) ───sqlplus (3894) ─┬─bash (3897) ───sqlplus (3910) ───oracle (3911)
│            │                                                                       └─oracle (3895)

[Email protected] ~]$ ps-ef |grep Oracle
can find
Oracle 3895 3894 0 17:34? 00:00:00 [Oracle]

One FG process turns into a zombie process

[Email protected] ~]$ Ps-o pid,ppid,state-p 3895
PID PPID S
3895 3894 Z

Kill-9 3895 doesn't kill the zombie process.

Workaround:

1. Can wait, will automatically end
sql> shutdown Immediate

Sql> Error Initializing Sql*plus
Internal Error
Hangup
[Email protected] ~]$ [[email protected] ~]$

2.kill parent process, child processes (even zombie) end together
Ps-al or Ps-o pid,ppid-p pid
Ppid is 3894.

Kill-9 3894

3.hang Live Terminal Try Ctrl + C, or CTRL + Z, or directly off, re-sqlplus/as SYSDBA
Shutdown abort
Startup restrict
Shutdown normal

Ctrl-c is sending a SIGINT signal to terminate a process
Ctrl-z is sending a sigstop signal, suspending a process

Ctrl-d does not send a signal. If the current line does not have an input character when ctrl-d, bash treats it as an EOF character, ending/exiting.
Delete the character after the cursor if the current line has input at ctrl-d. This is the ReadLine feature and does not send a signal.

/* The following references http://blog.csdn.net/leshami/article/details/9466559 */

Oracle's explanation for this is that this is not an Oracle bug, but instead uses an unhealthy database shutdown order.
It is recommended that you disconnect all connections before closing the database, such as turning off DB control first, see the following Oracle documentation and examples.

Doc ID 416658.1

Shutdown Immediate hangs/active Processes Prevent Shutdown (Doc ID 416658.1)

Applies To:
Oracle server-enterprise edition-version:10.1.0.2 to 11.2.0.2-release:10.1 to 11.2
Oracle server-enterprise edition-version:10.1.0.2 to 11.2.0.2 [release:10.1 to 11.2]
Information in this document applies to any platform.

Symptoms
The ' shutdown immediate ' command hangs or is very slow

The alert log lists messages like:

Shutdown:waiting for active calls to complete. ACTIVE PROCESSES PREVENT SHUTDOWN operation

Cause
This is not a bug.

If the DB Control repository is running on the database target against which shutdown immediate were attempted then an Inco Rrect
Order of events seems used.

Should stop DB control first to get rid of any connections between DB control and the repository database and then Shu Tdown
The database with ' shutdown immediate '.

Current database sessions may show:

Sql> Select SID, USERNAME, program from V$session;
SID USERNAME Program
----- ---------------------- ----------------------------------
243 SYSTEM SQL Developer
246 Sysman OMS
247[email protected](q001)
248[email protected](q000)
251 DBSNMP[email protected](TNS v1-v3)
252 Sysman OMS
253 Sysman OMS
254 Dbsnmp[email protected](TNS v1-v3)
255 SYSTEM Java.exe
Sysman OMS

Clearly OMS and OEM is connected (Oracle Enterprise Manager Grid Control or Dbconsole) via users Sysman and DBSNMP.
These sessions should be de-activated (that's to log off any OEM, OMS, Sysman and DBSNMP) before the shutdown immediate I S attempted.

Oracle Enterprise Manager, Grid Control, dbconsole and agents keep doing internal processing.
This could include a few plsql notification procedures running on the database by database control like
BEGIN emd_notification. Queue_ready (: 1,: 2,: 3); END;

As per internal documentation of the shutdown immediate, if there is active calls then it would wait for all the active C Alls to finish.

Solution
To implement the solution:

1. Given OEM Connections is active (Sysman and DBSNMP), de-activate these sessions, i.e. by stopping the Agent/dbconsole

2. Then shutdown immediate as normal

-OR-

There may processes still running and holding locks at the time a shutdown is issued.
Sometimes these is failed jobs or transactions, which is effectively ' zombies ', which is not able to receive a signal F Rom Oracle.

If This occurs, the only-to-shutdown the database is by doing:

Sql>
Shutdown abort
Startup restrict
Shutdown normal

The startup does any necessary recovery and cleanup, so this a valid cold backup can be taken afterward.

If this issue occurs frequently, it would is a good practice to see if there is any active user processes running in V$se Ssion or
V$process before shutting down the instance.

If the problem persists, and no apparent user processes is active, you can set this event prior to issuing the shutdown C Ommand
In order to see what is happening. This would dump a systemstate every 5 minutes while shutdown is hanging

Sql>
Connect/as SYSDBA
Alter session SET Events ' 10400 Trace name Context forever, Level 1 '

http://blog.itpub.net/28859270/viewspace-1067155/

Shutdown immediate when hang (reprint)