Oracle 11gR2 RAC database cannot be connected (Ora-12537:tns:connection closed) Resolution

Oracle 11gR2 RAC database cannot be connected (Ora-12537:tns:connection closed) Resolution

[Email protected] ~]$ Sqlplus/nolog

Sql*plus:release 11.2.0.1.0 Production on Wed 29 21:06:10 2012

Copyright (c) 1982, Oracle. All rights reserved.

Sql> Connect system/*** @astt
ERROR:
Ora-12537:tns:connection closed

[Email protected] app]# ls-ltr
Drwxr-xr-x 3 root oinstall 409608-28 19:05 grid
DRWXRWX---6 grid oinstall 409608-29 00:03 orainventory
Drwxrwxr-x 7 Oracle Oinstall 4096 08-29 00:20 Oracle
[Email protected] app]# chmod 755 Oracle
[Email protected] app]# chmod 755 grid
[Email protected] app]# ls-ltr
Drwxr-xr-x 3 root oinstall 409608-28 19:05 grid
DRWXRWX---6 grid oinstall 409608-29 00:03 orainventory
Drwxr-xr-x 7 Oracle Oinstall 4096 08-29 00:20 Oracle

[Email protected] ~]$ Sqlplus/nolog

Sql*plus:release 11.2.0.1.0 Production on Wed 29 21:28:232012

Copyright (c) 1982, Oracle. All rightsreserved.

Sql> Connect system/*** @astt;
????
Sql>

Sql> select instance_name from Gv$instance;

Instance_name
----------------
Astt1
Astt2

Sql> select instance_name from V$instance;

Instance_name
----------------
Astt1

Sql>

Reference:

Metalink [ID 1069517.1]:

Symptoms
In environment where listener home (including SCAN listener whichresides in Grid infrastructure/asm home) and database Hom e areowned by different OS user, ORA-12537 could happen when Connectingthrough listener, when creating database through DB CA, or wheninstalling database software and creating a database Inruninstaller. Job Role Separation is a typical example as SCAN andlocal Grid Home listener is owned differently than database.
Error detail12537, 00000, "Tns:connection closed"
*cause: "End of File" condition has been reached; Partner Hasdisconnected.
*action:none needed; This is an information message. Before the error is reported, connection could hang for awhile, and a core file may also be generated. Screen output:
Sql> Conn system/[email protected]
ERROR:
Ora-12537:tns:connection closedorprcr-1079:failed to start resource ora.db11.db
Crs-5011:check of resource Db11 failed:details at..
Crs-2674:start of Ora.db11.db on Racnode1 failed
Ora-12546:tns:permission denied
Listener_scann.log or Listener.log
Tns-12518:tns:listener could not hand off client connection
Tns-12546:tns:permission denied
Tns-12560:tns:protocol Adapter Error
Tns-00516:permission denied
Linux Error:13:permissiondeniedortns-12518:tns:listener could not hand off clientconnection
Tns-12547:tns:lost Contact
Tns-12560:tns:protocol Adapter Error
Tns-00517:lost Contact
Linux Error:32:brokenpipe
Runinstaller error if installinginfo:starting Output Reader Threads for Process/ocw/grid/bin/kfod
info:parsing KFOD-00300:OCI Error [-1] [OCI ERROR] [ora-12547:tns:lost contact
Info:parsing] [12547]
Info:parsing
Info:the Process/ocw/grid/bin/kfod exited with code 1
..
SEVERE: [FATAL] [INS-30502] No ASM disk group found.
Cause:there were no diskgroups managed by the ASM instance +asm1. Cause1. Listener owner (including SCAN Listener) can not accessoracle binary in database home:as Listener owner:
$ ls-l $RDBMS _home/bin/oracle
LS:/home/oracle/app/oracle/product/11.2/db/bin/oracle:permissiondenied
2. Oracle binary in database home have wrong permission:
Ls-l $RDBMS _home/bin/oracle
-rwxr-x--x 1 Oracle asmadmin 184286251 916:25/home/oracle/app/oracle/product/11.2/db/bin/oraclethe Permission "- Rwxr-x--x "is wrong as it ' s missing suid bit,oracle binary should has permission of 6751:
-rwsr-s--x 1 Oracle asmadmin 184286251 916:25/home/oracle/app/oracle/product/11.2/db/bin/oraclenote:if Job Role Sep Aration is in place, the group would be, otherwise it'll be
3. File System for database home does Setuid/suid Orhas Nosetuid/nosuid set:
Mount| Grep
/home/oracle on/dev/dsk/diskoracle Read/write/nosuid. Solution
Solution is to do sure file system for database home Hassetuid/suid set, database binary ($RDBMS _home/bin/oracle) Hascorr ECT ownership and permission, and listener owner is able toaccess database Oracle binary (as listener owner, "Ls-l$rdbms_ Home/bin/oracle "would tell" If it Job Role separation environment, please refer to thefollowing document:oracle? Grid Infrastructure
Installation Guide
11g Release 2 (11.2) Oracle ASM Job Role separation Option with Sysasm

The official explanations are as follows:

Applies To:
Oracle Net services-version:8.1.7.0to 11.2.0.1-release:8.1.7 to 11.2
Generic UNIX
Checked for relevance on 30-apr-2010
Symptoms
Connection to the database server fails with ORA-12546 orORA-12537

Local connections working fine but the connection via listenerfails with ORA-12537 or ORA-12547

[Email protected]]$ sqlplusmyuser/[email protected]

Sql*plus:release 11.1.0.7.0-production on Fri APR 30 21:32:182010

Copyright (c) 1982, the Oracle. All rights reserved.

ERROR:
Ora-12537:tns:connection closed

The listener log for the failing connection shows Followingerror message-

Tns-12518:tns:listener could not hand off clientconnection
Tns-12546:tns:permission denied
Tns-12560:tns:protocol Adapter Error
Tns-00516:permission denied
Linux Error:13:permission denied

Or

Tns-12518:tns:listener could not hand off clientconnection
Tns-12547:tns:lost Contact
Tns-12560:tns:protocol Adapter Error
Tns-00517:lost Contact
Ibm/aix RISC system/6000 Error:32:broken Pipe

Cause
Permissions on Oracle Binary ($ORACLE _home/bin/oracle) is NotSet correctly.

Solution
The OS level error message in the Listener log "Linux Error:13:permission denied" indicates that the listener is not ABL E to handoff the connection to Oracle processes or not able to spawn a newdedicated user process because of lack of permis Sion at the Oslevel. To resolve the error-

1. Check and correct the permission on/var/tmp/.oracledirectory. Permissions on the This directory should is 777 with stickybit set.

# chmod 01777/var/tmp/.oracle

2. Check and correct permissions on $ORACLE _home/bin/oraclebinary and make sure the it is set to 6751.

$ chmod 6751 $ORACLE _home/bin/oracle

Check your environment and do not see the right permissions

[Email protected] bin]# pwd
/oracle/product/11.2.0/bin
[Email protected] bin]# Ls-la Oracle
-rwxr-s--x 1 Oracle asmadmin 228886426 Jan 7 15:46 Oracle
[[email protected] bin]# chmod 6751 oracle--Note using root user privileges
[Email protected] bin]# Ls-la Oracle
-rwsr-s--x 1 Oracle asmadmin 228886426 Jan 7 15:46 Oracle

Modified after landing normal:

[Email protected] ~]$ Sqlplussys/[email PROTECTED]:1521/RACDBASSYSDBA

Sql*plus:release 11.2.0.2.0 Production on Thu Jan 17 14:48:252013

Copyright (c) 1982, Oracle. All rights reserved.

Connected to:
Oracle Database 11g Enterprise Edition Release 11.2.0.2.0-64bitproduction
With the partitioning, Real application Clusters, Automatic storagemanagement, OLAP,
Data Mining and Real application testing options

Sql>

Oracle 11gR2 RAC database cannot be connected (Ora-12537:tns:connection closed) Resolution