A DB instance in a 12C RAC automatically crash and reports errors such as ORA-27157, ORA-27300, etc.

Once a 12C RAC database is installed on rhel7.2, one of the DB instances is often automatically crash. Viewing the alert log found the following error message:

Errors in File/d12/app/oracle/diag/rdbms/rac12c/rac12c2/trace/rac12c2_j000_21047.trc:ora-27157:os post/wait Facility Removedora-27300:os System dependent OPERATION:SEMOP failed with Status:43ora-27301:os failure message:identi Fier removedora-27302:failure occurred At:sskgpwwait1fri Sep 16:50:53 2016Errors in file/d12/app/oracle/diag/rdbms/r Ac12c/rac12c2/trace/rac12c2_rmv0_20798.trc:ora-27157:os post/wait facility Removedfri Sep 16:50:53 2016Errors in File/d12/app/oracle/diag/rdbms/rac12c/rac12c2/trace/rac12c2_q005_21328.trc:ora-27157:os post/wait Facility Removedora-27300:os system dependent OPERATION:SEMOP failed with Status:43ora-27301:os failure Message:identifier Remo Vedora-27302:failure occurred at:sskgpwwait1

Error reason Description:

In rhel7.2, the Systemd-logind service introduces a new feature: Remove all IPC objects after a user exits the OS completely.
This attribute is controlled by the REMOVEIPC option in the/etc/systemd/logind.conf parameter file. For details, see Man logind.conf (5).

In rhel7.2, the default value for REMOVEIPC is yes

Therefore, when the last Oracle or grid user exits, the operating system will remove the user's shared memory segments and semaphores
Oracle ASM and Database's SGA need to use the shared memory segments, so the remove shared memory segments will crash the Oracle ASM and database Instances

Please refer to Redhat bug 1264533-https://bugzilla.redhat.com/show_bug.cgi?id=1264533

This issue affects all applications that use shared memory segments and semaphores, so both Oracle ASM instances and Oracle Database instances are affected.
oel7.2 to avoid this problem, explicitly set REMOVEIPC to No in the/etc/systemd/logind.conf configuration file.

The problem causes the phenomenon:

1) Installing 11.2 and 12c Gi/crs fails, because ASM crashes towards the end of the installation.2) upgrading to 11.2 and 12c Gi/crs fails.3) after Redhat Linux are upgraded to 7.2, 11.2 and 12c ASM and database instances crash.

Systemd-logind may remove the IPC object at any time, and the corresponding log phenomenon will be different when an error occurs. Like what:

Most common error This occurs is a following is found in the ASM or database Alert.log:ora-27157:os post/wait FACI  Lity Removedora-27300:os System dependent OPERATION:SEMOP failed with Status:43ora-27301:os failure message:identifier Removedora-27302:failure occurred at:sskgpwwait1

　　

The second observed error occurs during installation and upgrade when ASMCA fails with the following Error:kfod-00313:no ASM instances available. CSS Group services were successfully initilized by kgxgncinkfod-00105:could not open pfile ' [email protected] '

　　

The third observed error occurred during installation and upgrade:creation of ASM password file failed. Following error occurred:error in Process:/d12/app/12.1.0/grid/bin/orapwd Enter password for sys:opw-00009:could not es Tablish connection to Automatic Storage Management instance2015/11/20 21:38:45 clsrsc-184:configuration of ASM failed2015 /11/20 21:38:46 clsrsc-258:failed To configure and start ASM

　　

The fourth observed error is the following message was found in The/var/log/messages file around the time that ASM or data Base instance Crashed:nov 21:38:43 testc201 kernel:traps:oracle[24861] trap divide errorip:3896db8 sp:7ffef1de3c40 er ror:0 in oracle[400000+ef57000]

　　

Modification Method:

1). Set/etc/systemd/logind.conf in Removeipc=no
2). Restart the server or restart the Systemd-logind
Restart Systemd-logind:

# systemctl daemon-reload# systemctl Restart Systemd-logind

　　

MOS Doc:

Alert:setting Removeipc=yes on Redhat 7.2 crashes ASM and Database Instances as well as any application that Uses a Share D Memory Segment (SHM) or semaphores (SEM) (Doc ID 2081410.1)

A DB instance in a 12C RAC automatically crash and reports errors such as ORA-27157, ORA-27300, etc.