11g of OHASD cannot be started randomly

Install 11.2.0.1 grid under Oracle Linux Server Release 6.2, when installed, successfully completed (may be the last execution of the root.sh error, but did not notice), etc. after installing reboot host, found that CRS does not start. As a result, subsequent installation of the RAC does not work properly.

Reboot the host and see the CRS and Ohas processes at first, but the CRS process disappears and the OHASD process is only one.

The CRS process was first seen:

[Root@ol6-112-rac1 ~]# ps-ef |grep CRS
root 1983 1 0 10:39? 00:00:00/u01/app/11.2.0.3/grid/bin/crsctl.bin start has
root 2185 2153 1 10:40 pts/0 00:00:00 grep CRS
[Root@ol6-112-rac1 ~]#

If the process of strace CRS can be seen:

... nanosleep ({5, 0}, 0xbfb2a9b0) = 0 Open ("/U01/APP/11.2.0.3/GRID/CRS/MESG/CRSUS.MSB", o_rdonly) = 3 Fcntl64 (3, F_SETFD, F d_cloexec) = 0 Lseek (3, 0, Seek_set) = 0 Read (3, "\25\23\" \1\23\3\t\t\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0 " .., 256) = 256 Lseek (3, Seek_set) = 3, "\21 ' \0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0" -Lseek (3, 1024, seek_set) = 1024 Read (3, "\226\0\245\0\262\0\307\0\330\0\345\0\364\0\376\0\7\1\21\0013\1 \1d\1\231\1\303\1\370\1 "..., lseek) = 75264 (3,, seek_set) = 75264 Read (3," \3\0\32\20\3\0\32\0\33\20\0\0\30\1\ 34\20\0\0k\1\0\0\0\0|\1 CRSC "..., he) = Lseek (3, 115712, seek_set) = 115712 Read (3," \377\377\0\0\0\0\0\0\0\0\0\0\0\ 0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0 "..., lseek) = 116224 (3,, seek_set) = 116224 Read (3," \377\377\0\0\0\0\0\0\ 0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0 "..., lseek) = 116736 (3,, seek_set) = 116736 Read (3," \0\0\0\0\0\ 0\10\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0 "..., the same" = the 3 = 0 Fstat64 (1, {st_mode=s_ifchr|0600, St_rdev=makedev (5, 1), ...}) = 0 ioctl (1, SND Ctl_tmr_timebase or tcgets, {B38400-opost-isig-icanon-echo ...}) = 0 Mmap2 (NULL, 4096, prot_read| Prot_write, map_private| Map_anonymous,-1, 0) = 0xb55d6000 Write (1, "Crs-4124:oracle high Availabili" ...,) = Open ("/u01/app/11.2.0.3/grid/c RS/MESG/CRSUS.MSB ", o_rdonly) = 3 Fcntl64 (3, f_setfd, fd_cloexec) = 0 Lseek (3, 0, Seek_set) = 0 Read (3," \25\23\ "\1\23\3\t \t\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0 "..., 256) = 256 Lseek (3, seek_set) = Read (3," \21 "\0\0\0\0\0 \0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0 "..., lseek) = 1024 (3,, seek_set) = 1024 Read (3," \226\0\245\0 \262\0\307\0\330\0\345\0\364\0\376\0\7\1\21\0013\1;\1d\1\231\1\303\1\370\1 "..., lseek" = 3, 52224, SEEK_SET)
= 52224 Read (3, "\10\0\353\n\1\0008\0\354\n\1\0\212\0\355\n\3\0\320\0\356\n\0\0\24\1\240\17\0\0q\1" ..., 512) = 512 Lseek (3, 115712, seek_set) = 115712 READ (3, "\377\377\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0" ...,) = Lseek (3, 116224, SEEK_ SET) = 116224 Read (3, "\377\377\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0" ...,) = Lseek (3,
116736, Seek_set) = 116736 Read (3, "\0\0\0\0\0\0\10\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0" ..., 512) = 512 Close (3) = 0 Write (1, "Crs-4000:command Start failed," ...,) = Munmap (0xb55d7000, 35744) = 0 Futex (0XA05D2F4, Futex _cmp_requeue_private, 1, 2147483647, 0xa05d1d0, 4) = 1 Futex (0xa05d1d0, futex_wake_private, 1) = 1 Munmap (0xb4d51000, 4198
= 0 Munmap (0xb5152000, 143360) = 0 Munmap (0xb5175000, 143360) = 0 Munmap (0xb5198000, 143360) = 0 Exit_group (1) =?
 Process 1983 detached [Root@ol6-112-rac2 ~]#

Follow-up, the process disappears.

[Root@ol6-112-rac1 ~]# ps-ef |grep CRS
root 2248 2153 1 10:42 pts/0 00:00:00 grep CRS
[Root@ol6-112-rac1 ~]#

If Strace OHASD process, you can see:

[Oracle@ol6-112-rac1 cfgtoollogs]$ ps-ef |grep ohasd
root 2111 1 0 09:14? 00:00:00/u01/app/11.2.0.3/grid/bin/ohasd. Bin reboot
Root 2369 2155 0 09:31 pts/0 00:00:00./ohasd.bin start
Oracle 2400 2313 0 09:34 pts/1 00:00:00 grep OH ASD
[oracle@ol6-112-rac1 cfgtoollogs]$ strace-p 2369
attach:ptrace (Ptrace_attach, ...): Operation not Permitted
[Oracle@ol6-112-rac1 cfgtoollogs]$ exit
logout
[Root@ol6-112-rac1 ~]# 
[ Root@ol6-112-rac1 ~]# 
[root@ol6-112-rac1 ~]# strace-p 2369
Process 2369 to attached-interrupt
Open ("/VAR/TMP/.ORACLE/NPOHASD", o_wronly) =? Erestartsys (to be restarted)

we know that the boot order of 11g GI is (reference ID 1050908.1):

OHASD-->OHASD agents-->daemons (GIPCD, MDNSD, GPNPD,CTSSD, OCSSD, CRSD, EVMD asm etc), then CRSD-->CRSD Agents--> ; User resources (database, SCAN, listener etc)

We now see that the situation is not even OHASD started, do not see/ETC/INIT.D/INIT.OHASD run such a process.

Under normal circumstances, there should be:

Ps-ef|grep init.ohasd|grep-v grep
root 2279 1 0 18:14? 00:00:00/BIN/SH/ETC/INIT.D/INIT.OHASD Run

OK, let's try to start/etc/init.d/init.ohasd run manually:

Root@ol6-112-rac1 bin]#/etc/init.d/init.ohasd Run
mkfifo:cannot create FIFO '/VAR/TMP/.ORACLE/NPOHASD ': File exists
^c
[Root@ol6-112-rac1 bin]#

See that/VAR/TMP/.ORACLE/NPOHASD has been occupied, and we remove it:

[Root@ol6-112-rac1. oracle]# mv npohasd Npohasd.bak
[Root@ol6-112-rac1. oracle]#

And then it's going to start up smoothly. It seems that if you start ohasd manually, you can still get up, but why not start automatically.

Checking the Inittab is also true:

Cat/etc/inittab|grep init.ohasd
h1:35:respawn:/etc/init.d/init.ohasd run >/dev/null 2>&1 </dev/ Null

According to [ID 1050908.1], there is a sentence:

Note:oracle Linux 6 (OL6) or Red Hat Linux 6 (RHEL6) has deprecated inittab, rather, INIT.OHASD would be configured in
Upstart In/etc/init, however, the process "/etc/init.d/init.ohasd run" should still is up.
If any RC Snncommand script (located in RCN.D, example S98gcstartup) stuck, init process may not start
/etc/init.d/in IT.OHASD Run "; Please engage the OS vendor to find out why relevant Snncommand script stuck.

Checked, and sure enough, Linux 6 and 11.2.0.1 did not pass certified, after the certification is 11.2.0.3.

Under Linux 6 let 11.2.0.1 's ohasd start randomly, and you need to deal with it (if you are still in the process, perform steps starting from step 2nd before executing root.sh):

1. Roll back the root.sh execution first:

[Root@ol6-112-rac1 install]#./roothas.pl-deconfig-force-verbose 2013-06-06 14:53:07:checking for Super User privilege S 2013-06-06 14:53:07:user has super User privileges 2013-06-06 14:53:07:parsing The host name Using configuration param Eter file:./crsconfig_params CRS Resources for listeners are still configured crs-2613:could don't find resource ' ORA.CSSD
'.
Crs-4000:command Stop failed, or completed with errors.
Crs-2613:could not find resource ' ORA.CSSD '.
Crs-4000:command Delete failed, or completed with errors. crs-2791:starting shutdown of Oracle high availability services-managed the ' Ol6-112-rac1 ' crs-2673:attempting To stop ' ora.crsd ' in ' Ol6-112-rac1 ' crs-2790:starting shutdown of Cluster Ready services-managed resources on ' ol6-112-r Ac1 ' crs-2673:attempting to stop ' Ora. Listener_scan3.lsnr ' on ' Ol6-112-rac1 ' crs-2673:attempting to stop ' Ora. Data.dg ' on ' Ol6-112-rac1 ' crs-2673:attempting to stop ' Ora. Listener_scan2.lsnr ' on ' Ol6-112-rac1 ' Crs-2673:attempting to stop ' ora.ol6-112-rac2.vip ' in ' Ol6-112-rac1 ' crs-2673:attempting to stop ' Ora. Listener_scan1.lsnr ' on ' Ol6-112-rac1 ' crs-2673:attempting to stop ' ora.ol6-112-rac1.vip ' ' Ol6-112-rac1 ' crs-2677:st Op of ' ora.ol6-112-rac2.vip ' Ol6-112-rac1 ' succeeded Crs-2677:stop of ' ora.ol6-112-rac1.vip ' on ' Ol6-112-rac1 ' Succee ded crs-2677:stop of ' Ora.  Listener_scan3.lsnr ' on ' Ol6-112-rac1 ' succeeded crs-2673:attempting to stop ' ora.scan3.vip ' on ' Ol6-112-rac1 ' CRS-2677: Stop ' Ora.scan3.vip ' on ' Ol6-112-rac1 ' succeeded Crs-2677:stop of ' Ora.  Listener_scan2.lsnr ' on ' Ol6-112-rac1 ' succeeded crs-2673:attempting to stop ' ora.scan2.vip ' on ' Ol6-112-rac1 ' CRS-2677: Stop ' Ora.scan2.vip ' on ' Ol6-112-rac1 ' succeeded Crs-2677:stop of ' Ora.  Listener_scan1.lsnr ' on ' Ol6-112-rac1 ' succeeded crs-2673:attempting to stop ' ora.scan1.vip ' on ' Ol6-112-rac1 ' CRS-2677: Stop ' Ora.scan1.vip ' on ' Ol6-112-rac1 ' succeeded Crs-2677:stop of ' Ora. Data.dg ' on ' Ol6-112-rac1 ' succeeded Crs-2673:aTtempting to stop ' ora.asm ' in ' Ol6-112-rac1 ' crs-2677:stop of ' ora.asm ' on ' Ol6-112-rac1 ' succeeded  To stop ' ora.eons ' in ' Ol6-112-rac1 ' crs-2673:attempting to stop ' ora.ons ' ol6-112-rac1 ' crs-2677:stop ' ora.ons ' On ' Ol6-112-rac1 ' succeeded crs-2673:attempting to stop ' ora.net1.network ' ol6-112-rac1 ' crs-2677:stop of ' Ora.net1 . Network ' on ' Ol6-112-rac1 ' succeeded Crs-2677:stop of ' ora.eons ' on ' Ol6-112-rac1 ' succeeded Crs-2792:shutdown of Clust ER Ready services-managed resources to ' OL6-112-RAC1 ' has completed crs-2677:stop of ' ora.crsd ' on ' Ol6-112-rac1 ' succeed Ed crs-2673:attempting to stop ' ora.gpnpd ' in ' Ol6-112-rac1 ' crs-2673:attempting to stop ' ora.cssdmonitor ' on ' ol6-112-r
Ac1 ' crs-2673:attempting to stop ' ora.ctssd ' in ' Ol6-112-rac1 ' crs-2673:attempting to stop ' ora.evmd ' ' Ol6-112-rac1 ' Crs-2673:attempting to stop ' ora.asm ' in ' Ol6-112-rac1 ' crs-2673:attempting to stop ' ora.mdnsd ' on ' Ol6-112-rac1 ' CRS-26 77:stop of ' Ora.cssdmonItor ' on ' Ol6-112-rac1 ' succeeded Crs-2677:stop of ' ora.gpnpd ' on ' Ol6-112-rac1 ' succeeded Crs-2677:stop of ' ORA.EVMD ' O N ' Ol6-112-rac1 ' succeeded Crs-2677:stop of ' ora.mdnsd ' on ' Ol6-112-rac1 ' succeeded Crs-2677:stop's ' ORA.CTSSD ' ' on ' ' ol6 -112-rac1 ' succeeded Crs-2677:stop of ' ora.asm ' ' ol6-112-rac1 ' succeeded ' crs-2673:attempting to Stop ' ora.cssd ' on ' O L6-112-rac1 ' crs-2677:stop of ' ora.cssd ' on ' Ol6-112-rac1 ' succeeded crs-2673:attempting to Stop ' Ora.diskmon ' on ' ol6-1 12-rac1 ' crs-2673:attempting to stop ' ora.gipcd ' in ' Ol6-112-rac1 ' crs-2677:stop of ' ora.gipcd ' on ' Ol6-112-rac1 ' Succee Ded crs-2677:stop of ' Ora.diskmon ' on ' Ol6-112-rac1 ' succeeded Crs-2793:shutdown-Oracle high Availability Services-ma
naged resources in ' OL6-112-RAC1 ' has completed crs-4133:oracle high availability Services has been. ADVM/ACFS is isn't supported on oraclelinux-release-6server-2.0.2.i686 Acfs-9201:not supported Successfully deconfigured O racle Restart Stack [root@ol6-112-raC1 install]# 

2. Modify the $gi_home/install/s_crsconfig_lib.pm and add the following code to the # Start OHASD:

vi/u01/app/11.2.0.3/grid/crs/install/s_crsconfig_lib.pm ...
Sleep (5);
S_add_itab () or return $FAILED;
System ("$INIT Q");


# # = Add by oracleblog.org. Start here = = I
$UPSTART _ohasd_service = "ORACLE-OHASD";
My $INITCTL = "/sbin/initctl";

($status, @output) = System_cmd_capture ("$INITCTL start $UPSTART _ohasd_service");
if (0!= $status)
{
error ("Failed to start $UPSTART _ohasd_service, Error: $!");
return $FAILED;
}
# # = Add by oracleblog.org. end here = =

# Start ohasd

$status = System ("$CRSCTL start has");
......

3. Create a/etc/init/oracle-ohasd.conf file with the following contents:

# Oracle OHASD startup

start on runlevel [km]
stop on RunLevel [!35]
respawn
exec/etc/init.d/ INIT.OHASD Run >/dev/null 2>&1 </dev/null

4. Run root.sh again

[Root@ol6-112-rac1 grid]# sh root.sh Running Oracle 11g root.sh script ... The following environment variables are set as:oracle_owner= ORACLE oracle_home=/u01/app/11.2.0.3/grid Enter the full p Athname of the Local bin directory: [/usr/local/bin]: The file "Dbhome" already exists in/usr/local/bin. Overwrite it? 
(y/n)
[n]: y copying dbhome to/usr/local/bin ... The file "Oraenv" already exists in/usr/local/bin. Overwrite it? 
(y/n)
[n]: y copying oraenv to/usr/local/bin ... The file "Coraenv" already exists in/usr/local/bin. Overwrite it? 
(y/n)

[n]: y copying coraenv to/usr/local/bin ... Entries'll be added to The/etc/oratab file as needed by database Configuration Assistant when a The database is created Fin
Ished running generic part of root.sh script.
Now product-specific root actions would be performed. 2013-06-06 15:04:23:parsing the host name 2013-06-06 15:04:23:checking for Super User privileges 2013-06-06 ER has super user privileges Using configuration paRameter file:/u01/app/11.2.0.3/grid/crs/install/crsconfig_params local ADD MODE creating OCR keys for user ' root ', PRIVG
RP ' Root '.
Operation successful.
Adding daemon to Inittab crs-4123:oracle high availability Services has been. OHASD is starting ADVM/ACFS isn't supported on oraclelinux-release-6server-2.0.2.i686 crs-2672:attempting to start ' Ora . gipcd ' on ' Ol6-112-rac1 ' crs-2672:attempting to start ' ora.mdnsd ' ' ol6-112-rac1 ' crs-2676:start ' of ' ora.gipcd ' ' O ' L6-112-rac1 ' succeeded Crs-2676:start of ' ora.mdnsd ' ' Ol6-112-rac1 ' succeeded crs-2672:attempting to Start ' ORA.GPNPD ' On ' ol6-112-rac1 ' Crs-2676:start of ' ora.gpnpd ' in ' Ol6-112-rac1 ' succeeded crs-2672:attempting to Start ' Ora.cssdmoni Tor ' on ' Ol6-112-rac1 ' Crs-2676:start of ' ora.cssdmonitor ' in ' Ol6-112-rac1 ' succeeded crs-2672:attempting to Start ' Ora  . CSSD ' on ' Ol6-112-rac1 ' crs-2672:attempting to start ' Ora.diskmon ' ' ol6-112-rac1 ' crs-2676:start ' ' Ora.diskmon ' on ' Ol6-112-rac1 ' succeeded
Crs-2676:start of ' ora.cssd ' on ' Ol6-112-rac1 ' succeeded Crs-2500:cannot stop resource ' ORA.CRSD ' as it's not running
Crs-4000:command Stop failed, or completed with errors. Command return code of 1 (256) from command:/u01/app/11.2.0.3/grid/bin/crsctl stop resource ora.crsd-init stop of Resour Ce "Ora.crsd-init" failed failed to stop CRSD crs-2500:cannot stop resource ' ora.asm ' as it's not running crs-4000:com
Mand Stop failed, or completed with errors. Command return code of 1 (256) from command:/u01/app/11.2.0.3/grid/bin/crsctl stop resource ora.asm-init stop of RESOURC E "Ora.asm-init" failed failed to stop ASM crs-2500:cannot stop resource ' ORA.CTSSD ' as it's not running Crs-4000:comm
And Stop failed, or completed with errors. Command return code of 1 (256) from command:/u01/app/11.2.0.3/grid/bin/crsctl stop resource ora.ctssd-init stop of Resou Rce "Ora.ctssd-init" failed failed to stop OCTSSD crs-2673:attempting to stop ' ora.cssdmonitor ' on ' Ol6-112-rac1 ' CRS-26 77: Stop ' ora.cssdmonitor ' on ' Ol6-112-rac1 ' succeeded crs-2673:attempting to stop ' ora.cssd ' on ' Ol6-112-rac1 ' CRS-2677  : Stop ' ora.cssd ' on ' Ol6-112-rac1 ' succeeded crs-2673:attempting to stop ' ora.gpnpd ' on ' Ol6-112-rac1 ' Crs-2677:stop of ' ora.gpnpd ' on ' Ol6-112-rac1 ' succeeded crs-2673:attempting to stop ' ora.gipcd ' ' ol6-112-rac1 ' crs-2677:stop ' Ora.gipcd ' on ' Ol6-112-rac1 ' succeeded crs-2673:attempting to stop ' ora.mdnsd ' on ' ol6-112-rac1 ' crs-2677:stop of ' ora.m DNSD ' on ' OL6-112-RAC1 ' succeeded Initial cluster configuration. See/u01/app/11.2.0.3/grid/cfgtoollogs/crsconfig/rootcrs_ol6-112-rac1.log for details [Root@ol6-112-rac1 grid]#

After the reboot, you find that the OHASD is ready to start randomly.

# #原文链接: http://www.oracleblog.org/working-case/ohasd-can-not-auto-start-with-server-reboot/