11.2.0.3LinuxRAC error CRS-5018 :(: CLSN00037 :) Removedunused

11.2.0.3LinuxRAC error CRS-5018 :(: CLSN00037 :) RemovedunusedHAIProute:

11.2.0.3 Linux RAC error CRS-5018 :(: CLSN00037 :) Removed unused HAIP route:

One of the nodes in a payment system 11.2.0.3 cannot be started.

1. Try to close the cluster and restart the Cluster

[Root @ rac2 ~] $ Crsctl stop crs-f

2. Try restarting the Cluster

[Root @ rac2 ~] $ Crsctl start crs

CRS-4123: Oracle High Availability Services has been started.

The cluster is successfully started and no other errors are reported. At this time, the asm is ready.

[Grid @ rac2 ~] $ Ps-ef | grep smon

Root 8472 1 1? 00:00:01/u01/app/11.2/product/crs_1/bin/osysmond. bin

Grid 9238 1 0? 00:00:00 asm_smon _ + ASM2

Grid 9500 6212 0 00:00:00 pts/5 grep smon

At this time, I tried to connect to ASM to check the status of dg. A strange thing happened. asm can log on, but an error is returned when I query asmdisk.

[Grid @ rac2 ~] $ Sqlplus/as sysasm

SQL * Plus: Release 11.2.0.3.0 Production on Wed May 21 15:35:05 2014

Connected:

Oracle Database 11g Enterprise Edition Release 11.2.0.3.0-64bit Production

With the Real Application Clusters and Automatic Storage Management options

SQL> select status from v $ asm_disk;

Select status from v $ asm_disk

*

ERROR at line 1:

A ORA-01034: ORACLE not available

Process ID: 14465

Session ID: 2707 Serial number: 3

Check the d. bin process and find that crsd is not up.

[Root @ rac01 ~] # Ps-ef | grep d. bin

Root 4142 1 0 09:27? 00:00:05/u01/app/11.2.0/grid/product/db_1/bin/ohasd. bin reboot

Grid 4548 1 0? 00:00:00/u01/app/11.2.0/grid/product/db_1/bin/mdnsd. bin

Grid 4558 1 0? 00:00:01/u01/app/11.2.0/grid/product/db_1/bin/gpnpd. bin

Grid 4568 1 0? 00:00:05/u01/app/11.2.0/grid/product/db_1/bin/gipcd. bin

Root 4590 1 0 09:28? 00:00:10/u01/app/11.2.0/grid/product/db_1/bin/osysmond. bin

Check whether the asm process exists again.

[Grid @ rac2 ~] $ Ps-ef | grep smon

Root 17377 1 2? 00:00:09/u01/app/11.2/product/crs_1/bin/osysmond. bin

Grid 21518 1 0 14: 37? 00:00:00 asm_smon _ + ASM2

Grid 21615 18834 0 00:00:00 pts/5 grep smon

-------------------------------------- Split line --------------------------------------

Install Oracle 11gR2 (x64) in CentOS 6.4)

Steps for installing Oracle 11gR2 in vmwarevm

Install Oracle 11g XE R2 In Debian

-------------------------------------- Split line --------------------------------------

Logging on to asmcmd also throws an exception.

[Grid @ rac2 ~] $ Asmcmd

A ORA-01034: ORACLE not available

Process ID: 21716

Session ID: 2707 Serial number: 1 (dbd error: OCIStmtExecute/Describe)

Normal check failed for ocr

[Root @ rac2 bin] #./ocrcheck

PROT-602: Failed to retrieve data from the cluster registry

PROC-26: Error while accessing the physical storage

ORA-15077: cocould not locate ASM instance serving a required diskgroup

Close asm and try to start the asm instance manually. Similarly, the asm instance cannot be started.

[Grid @ rac2 ~] $ Sqlplus/as sysasm

SQL * Plus: Release 11.2.0.3.0 Production on Wed May 21 14:33:37 2014

Copyright (c) 1982,201 1, Oracle. All rights reserved.

SQL> shutdown abort

ASM instance shutdown

SQL> startup

ORA-27103: internal error

Linux-x86_64 Error: 2: No such file or directory

Additional information: 1

Additional information: 25919497

Additional information: 2

Create a new pfile through the spfile of the + ASM1 instance and try to start the + ASM2 instance. Similarly, the + ASM2 instance cannot be started.

SQL> startup nomount pfile = '/tmp/init + asm2.ora ';

The ORA-24324: service handle not initialized.

ORA-01041: internal error. hostdef extension doesn't exist

At the beginning of the year, it was suspected that it was a storage problem, so we began to check the storage status. This RAC uses the multipath + asmlib architecture.

Status of multipath query, all active

[Root @ rac2 bin] # multipath-ll

360a9800044336b327a24446172587864 dm-5 NETAPP, LUN

[Size = 200G] [features = 3 queue_if_no_path pg_init_retries 50] [hwhandler = 1 alua] [rw]

\ _ Round-robin 0 [prio = 50] [active]

\ _ 4: 0: 1: 0 sdau 66: 224 [active] [ready]

\ _ 3: 0: 1: 0 sdq 65: 0 [active] [ready]

\ _ Round-robin 0 [prio = 10] [enabled]

\ _ 4: 0: 0: 0 sdaf 65: 240 [active] [ready]

\ _ 3: 0: 0: 0 sdb 8: 16 [active] [ready]

360a9800044336b327a24446172587862 dm-6 NETAPP, LUN

[Size = 200G] [features = 3 queue_if_no_path pg_init_retries 50] [hwhandler = 1 alua] [rw]

\ _ Round-robin 0 [prio = 50] [active]

\ _ 4: 0: 1: 1 sdav 66: 240 [active] [ready]

\ _ 3: 0: 1: 1 sdr 65: 16 [active] [ready]

\ _ Round-robin 0 [prio = 10] [enabled]

\ _ 4: 0: 0: 1 sdag 66: 0 [active] [ready]

\ _ 3: 0: 0: 1 sdc 8: 32 [active] [ready]

360a9800044336b327a2444617258786e dm-13 NETAPP, LUN

[Size = 200G] [features = 3 queue_if_no_path pg_init_retries 50] [hwhandler = 1 alua] [rw]

\ _ Round-robin 0 [prio = 50] [active]

\ _ 4: 0: 1: 8 sdbc 67: 96 [active] [ready]

\ _ 3: 0: 1: 8 sdy 65: 128 [active] [ready]

\ _ Round-robin 0 [prio = 10] [enabled]

\ _ 4: 0: 0: 8 sdan 66: 112 [active] [ready]

\ _ 3: 0: 0: 8 sdj 8:144 [active] [ready]

360a9800044336b327a24446172587876 dm-11 NETAPP, LUN

[Size = 2.0G] [features = 3 queue_if_no_path pg_init_retries 50] [hwhandler = 1 alua] [rw]

\ _ Round-robin 0 [prio = 50] [active]

\ _ 4: 0: 1: 6 sdba 67: 64 [active] [ready]

\ _ 3: 0: 1: 6 sdw 65: 96 [active] [ready]

\ _ Round-robin 0 [prio = 10] [enabled]

\ _ 4: 0: 0: 6 sdal 66: 80 [active] [ready]

\ _ 3: 0: 0: 6 sdh 8:112 [active] [ready]

[Size = 10G] [features = 3 queue_if_no_path pg_init_retries 50] [hwhandler = 1 alua] [rw]

\ _ Round-robin 0 [prio = 50] [active]

\ _ 3: 0: 1: 13 sdac 65: 192 [active] [ready]

\ _ 4: 0: 1: 13 sdbg 67: 160 [active] [ready]

\ _ Round-robin 0 [prio = 10] [enabled]

\ _ 4: 0: 0: 13 sdar 66: 176 [active] [ready]

\ _ 3: 0: 0: 13 sdn 8:208 [active] [ready]

360a9800044336b327a24446172587868 dm-0 NETAPP, LUN

[Size = 200G] [features = 3 queue_if_no_path pg_init_retries 50] [hwhandler = 1 alua] [rw]

\ _ Round-robin 0 [prio = 50] [active]

\ _ 3: 0: 1: 10 sdaa 65: 160 [active] [ready]

\ _ 4: 0: 1: 10 sdbe 67: 128 [active] [ready]

\ _ Round-robin 0 [prio = 10] [enabled]

\ _ 4: 0: 0: 10 sdap 66: 144 [active] [ready]

\ _ 3: 0: 0: 10 sdl 8: 176 [active] [ready]

360a9800044336b327a24446172587870 dm-9 NETAPP, LUN

[Size = 2.0G] [features = 3 queue_if_no_path pg_init_retries 50] [hwhandler = 1 alua] [rw]

\ _ Round-robin 0 [prio = 50] [active]

\ _ 4: 0: 1: 4 sday 67: 32 [active] [ready]

\ _ 3: 0: 1: 4 sdu 65: 64 [active] [ready]

\ _ Round-robin 0 [prio = 10] [enabled]

\ _ 4: 0: 0: 4 sdaj 66: 48 [active] [ready]

\ _ 3: 0: 0: 4 sdf 8: 80 [active] [ready]

360a9800044336b327a24446172587970 dm-1 NETAPP, LUN

\ _ Round-robin 0 [prio = 50] [active]

\ _ 3: 0: 1: 15 sdae 65: 224 [active] [ready]

\ _ 4: 0: 1: 15 sdbi 67: 192 [active] [ready]

\ _ Round-robin 0 [prio = 10] [enabled]

\ _ 4: 0: 0: 15 sdat 66: 208 [active] [ready]

\ _ 3: 0: 0: 15 sdp 8:240 [active] [ready]

360a9800044336b327a24446172587a55 dm-3 NETAPP, LUN

[Size = 5.0G] [features = 3 queue_if_no_path pg_init_retries 50] [hwhandler = 1 alua] [rw]

\ _ Round-robin 0 [prio = 50] [active]

\ _ 3: 0: 1: 14 sdad 65: 208 [active] [ready]

\ _ 4: 0: 1: 14 sdbh 67: 176 [active] [ready]

\ _ Round-robin 0 [prio = 10] [enabled]

\ _ 4: 0: 0: 14 sdas 66: 192 [active] [ready]

\ _ 3: 0: 0: 14 sdo 8:224 [active] [ready]

360a9800044336b327a2444617258786a dm-8 NETAPP, LUN

[Size = 200G] [features = 3 queue_if_no_path pg_init_retries 50] [hwhandler = 1 alua] [rw]

\ _ Round-robin 0 [prio = 50] [active]

\ _ 4: 0: 1: 3 sdax 67: 16 [active] [ready]

\ _ 3: 0: 1: 3 sdt 65: 48 [active] [ready]

\ _ Round-robin 0 [prio = 10] [enabled]

\ _ 4: 0: 0: 3 sdai 66: 32 [active] [ready]

\ _ 3: 0: 0: 3 sde 8: 64 [active] [ready]

360a9800044336b327a24446172587872 dm-14 NETAPP, LUN

[Size = 200G] [features = 3 queue_if_no_path pg_init_retries 50] [hwhandler = 1 alua] [rw]

\ _ Round-robin 0 [prio = 50] [active]

\ _ 4: 0: 1: 9 sdbd 67: 112 [active] [ready]

\ _ 3: 0: 1: 9 sdz 65: 144 [active] [ready]

\ _ Round-robin 0 [prio = 10] [enabled]

\ _ 4: 0: 0: 9 sdao 66: 128 [active] [ready]

\ _ 3: 0: 0: 9 sdk 8: 160 [active] [ready]

360a9800044336b327a24446172587972 dm-2 NETAPP, LUN

[Size = 10G] [features = 3 queue_if_no_path pg_init_retries 50] [hwhandler = 1 alua] [rw]

\ _ Round-robin 0 [prio = 50] [active]

\ _ 3: 0: 1: 13 sdac 65: 192 [active] [ready]

\ _ 4: 0: 1: 13 sdbg 67: 160 [active] [ready]

\ _ Round-robin 0 [prio = 10] [enabled]

\ _ 4: 0: 0: 13 sdar 66: 176 [active] [ready]

\ _ 3: 0: 0: 13 sdn 8:208 [active] [ready]

360a9800044336b327a24446172587868 dm-0 NETAPP, LUN

[Size = 200G] [features = 3 queue_if_no_path pg_init_retries 50] [hwhandler = 1 alua] [rw]

\ _ Round-robin 0 [prio = 50] [active]

\ _ 3: 0: 1: 10 sdaa 65: 160 [active] [ready]

\ _ 4: 0: 1: 10 sdbe 67: 128 [active] [ready]

\ _ Round-robin 0 [prio = 10] [enabled]

\ _ 4: 0: 0: 10 sdap 66: 144 [active] [ready]

\ _ 3: 0: 0: 10 sdl 8: 176 [active] [ready]

The asm disk in asmlib can also be properly identified. The storage fault is eliminated and the grid log is located.

[Root @ rac2 bin] #/etc/init. d/multipathd restart

Stopping multipathd daemon: [OK]

Starting multipathd daemon: [OK]

[Root @ rac2 bin] #/etc/init. d/oracleasm restart

Dropping Oracle ASMLib disks: [OK]

Shutting down the Oracle ASMLib driver: [OK]

Initializing the Oracle ASMLib driver: [OK]

Scanning the system for Oracle ASMLib disks: [OK]

[Root @ rac2 bin] # oracleasm scandisks

Reloading disk partitions: done

Cleaning any stale ASM disks...

Scanning system for ASM disks...

[Root @ rac2 bin] # oracleasm listdisks

ASMLB_OCR01

ASMLB_OCR02

ASMLB_OCR03

ASMLB_ORCL01

ASMLB_ORCL02

ASMLB_ORCL03

ASMLB_ORCLCTL

ASMLB_ORCLFRA

ASMLB_ORCLREDO

ASMLB_YYZF01

ASMLB_YYZF02

ASMLB_YYZF03

ASMLB_YYZFCTL

ASMLB_YYZFFRA

ASMLB_YYZREDO

After the crs is restarted, it is reported that the system still has the ASM process, but the asm cannot mount the disk, and the crsd still cannot start.

Alerrac2.log logs are as follows:

[/U01/app/11.2/product/crs_1/bin/orarootagent. bin (13382)] CRS-5018 :(: CLSN00037 :) Removed unused HAIP route: 169.254.95.0/255.255.255.0/0.0.0.0/usb0

13:18:09. 087

[/U01/app/11.2/product/crs_1/bin/orarootagent. bin (13382)] CRS-5018 :(: CLSN00037 :) Removed unused HAIP route: 169.254.95.0/255.255.255.0/0.0.0.0/usb0

2014-05-21 13:22:37. 716

[Ctssd (17298)] CRS-2409: The clock on host rac2 is not synchronous with the mean cluster time. No action has been taken as the Cluster Time Synchronization Service is running in observer mode.

2014-05-21 13:38:09. 677

[/U01/app/11.2/product/crs_1/bin/orarootagent. bin (13382)] CRS-5018 :(: CLSN00037 :) Removed unused HAIP route: 169.254.95.0/255.255.255.0/0.0.0.0/usb0

2014-05-21 13:52:38. 315

[Ctssd (17298)] CRS-2409: The clock on host rac2 is not synchronous with the mean cluster time. No action has been taken as the Cluster Time Synchronization Service is running in observer mode.

2014-05-21 13:58:10. 245

[/U01/app/11.2/product/crs_1/bin/orarootagent. bin (13382)] CRS-5018 :(: CLSN00037 :) Removed unused HAIP route: 169.254.95.0/255.255.255.0/0.0.0.0/usb0

[Gpnpd (8438)] CRS-2328: GPNPD started on node rac2.

2014-05-21 14:14:29. 618

[Cssd (8520)] CRS-1713: CSSD daemon is started in clustered mode

2014-05-21 14:14:31. 419

[Ohasd (8217)] CRS-2767: Resource state recovery not attempted for 'ora. diskmon' as its target state is OFFLINE

2014-05-21 14:14:50. 409

[Cssd (8520)] CRS-1707: Lease acquisition for node rac2 number 2 completed

2014-05-21 14:14:51. 693

[Cssd (8520)] CRS-1621: The IPMI configuration data for this node stored in the Oracle registry is incomplete; details (: CSSNK00002 :) in/u01/app/11.2/product/crs_1/log/rac2/cssd/ocssd. log

2014-05-21 14:14:51. 693

[Cssd (8520)] CRS-1617: The information required to do node kill for node rac2 is incomplete; details (: CSSNM00004 :) in/u01/app/11.2/product/crs_1/log/rac2/cssd/ocssd. log

2014-05-21 14:14:51. 696

[Cssd (8520)] CRS-1605: CSSD voting file is online: ORCL: ASMLB_OCR03; details in/u01/app/11.2/product/crs_1/log/rac2/cssd/ocssd. log.

2014-05-21 14:14:51. 700

[Cssd (8520)] CRS-1605: CSSD voting file is online: ORCL: ASMLB_OCR02; details in/u01/app/11.2/product/crs_1/log/rac2/cssd/ocssd. log.

2014-05-21 14:14:51. 704

[Cssd (8520)] CRS-1605: CSSD voting file is online: ORCL: ASMLB_OCR01; details in/u01/app/11.2/product/crs_1/log/rac2/cssd/ocssd. log.

2014-05-21 14:14:56. 235

[Cssd (8520)] CRS-1601: CSSD Reconfiguration complete. Active nodes are rac2 rac2.

2014-05-21 14:14:58. 695

[Ctssd (8722)] CRS-2403: The Cluster Time Synchronization Service on host rac2 is in observer mode.

2014-05-21 14:14:58. 973

[Ctssd (8722)] CRS-2407: The new Cluster Time Synchronization Service reference node is host rac2.

2014-05-21 14:14:58. 974

[Ctssd (8722)] CRS-2401: The Cluster Time Synchronization Service started on host rac2.

[Client (8758)] CRS-10001: 21-May-14 ACFS-9391: Checking for existing ADVM/ACFS installation.

[Client (8763)] CRS-10001: 21-May-14 ACFS-9392: Validating ADVM/ACFS installation files for operating system.

[Client (8765)] CRS-10001: 21-May-14 ACFS-9393: Verifying ASM Administrator setup.

[Client (8768)] CRS-10001: 21-May-14 ACFS-9308: Loading installed ADVM/ACFS drivers.

[Client (8771)] CRS-10001: 21-May-14 ACFS-9154: Loading 'oracleoks. ko 'driver.

[Client (8812)] CRS-10001: 21-May-14 ACFS-9154: Loading 'oracleadvm. ko 'driver.

[Client (8854)] CRS-10001: 21-May-14 ACFS-9154: Loading 'oracleacfs. ko 'driver.

[Client (8966)] CRS-10001: 21-May-14 ACFS-9327: Verifying ADVM/ACFS devices.

[Client (8971)] CRS-10001: 21-May-14 ACFS-9156: Detecting control device '/dev/asm/. asm_ctl_spec '.

[Client (8975)] CRS-10001: 21-May-14 ACFS-9156: Detecting control device '/dev/ofsctl '.

[Client (8981)] CRS-10001: 21-May-14 ACFS-9322: completed

2014-05-21 14:26:29. 083

[/U01/app/11.2/product/crs_1/bin/orarootagent. bin (14037)] CRS-5018 :(: CLSN00037 :) Removed unused HAIP route: 169.254.95.0/255.255.255.0/0.0.0.0/usb0

[Cssd (14105)] CRS-1612: Network communication with node rac2 (1) missing for 50% of timeout interval. Removal of this node from cluster in 14.010 seconds

2014-05-21 14:28:40. 898

[Cssd (14105)] CRS-1662: Member kill requested by node rac2 for member number 1, group DB + ASM

2014-05-21 14:28:42. 637

[/U01/app/11.2/product/crs_1/bin/oraagent. bin (13997)] CRS-5019: All OCR locations are on ASM disk groups [OCRDG],

And none of these disk groups are mounted. Details are at "(: CLSN00100 :)"

In "/u01/app/11.2/product/crs_1/log/rac2/agent/ohasd/oraagent_grid/oraagent_grid.log ".

2014-05-21 14:28:42. 637

[/U01/app/11.2/product/crs_1/bin/oraagent. bin (13997)] CRS-5011: Check of resource "+ ASM" failed: details at "(: CLSN00006 :)"

In "/u01/app/11.2/product/crs_1/log/rac2/agent/ohasd/oraagent_grid/oraagent_grid.log"

2014-05-21 14:28:42. 919

[/U01/app/11.2/product/crs_1/bin/oraagent. bin (13997)] CRS-5019:

All OCR locations are on ASM disk groups [OCRDG], and none of these disk groups are mounted. Details are at "(: CLSN00100 :)"

In "/u01/app/11.2/product/crs_1/log/rac2/agent/ohasd/oraagent_grid/oraagent_grid.log ".

2014-05-21 14:28:42. 920

[/U01/app/11.2/product/crs_1/bin/oraagent. bin (13997)]

CRS-5011: Check of resource "+ ASM" failed: details at "(: CLSN00006 :)"

In "/u01/app/11.2/product/crs_1/log/rac2/agent/ohasd/oraagent_grid/oraagent_grid.log"

2014-05-21 14:28:43. 147

[/U01/app/11.2/product/crs_1/bin/oraagent. bin (13997)] CRS-5019

: All OCR locations are on ASM disk groups [OCRDG], and none of these disk groups are mounted.

Details are at "(: CLSN00100 :)" in "/u01/app/11.2/product/crs_1/log/rac2/agent/ohasd/oraagent_grid/oraagent_grid.log ".

2014-05-21 14:28:43. 147

[/U01/app/11.2/product/crs_1/bin/oraagent. bin (13997)]

CRS-5011: Check of resource "+ ASM" failed: details at "(: CLSN00006 :)"

In "/u01/app/11.2/product/crs_1/log/rac2/agent/ohasd/oraagent_grid/oraagent_grid.log"

2014-05-21 14:28:43. 429

[/U01/app/11.2/product/crs_1/bin/oraagent. bin (13997)]

CRS-5019: All OCR locations are on ASM disk groups [OCRDG], and none of these disk groups are mounted.

Details are at "(: CLSN00100 :)" in "/u01/app/11.2/product/crs_1/log/rac2/agent/ohasd/oraagent_grid/oraagent_grid.log ".

2014-05-21 14:28:43. 430

[/U01/app/11.2/product/crs_1/bin/oraagent. bin (13997)] CRS-5011: Check of resource "+ ASM" failed: details at "(: CLSN00006 :)"

In "/u01/app/11.2/product/crs_1/log/rac2/agent/ohasd/oraagent_grid/oraagent_grid.log"

2014-05-21 14:30:44. 648

[Cssd (14105)] CRS-1662: Member kill requested by node rac2 for member number 1, group DB + ASM

2014-05-21 14:30:47. 200

[/U01/app/11.2/product/crs_1/bin/oraagent. bin (13997)] CRS-5019: All OCR locations are on ASM disk groups [OCRDG],

And none of these disk groups are mounted. details are at "(: CLSN00100 :)" in "/u01/app/11.2/product/crs_1/log/rac2/agent/ohasd/oraagent_grid/oraagent_grid.log ".

2014-05-21 14:30:47. 201