/limits.conf Oracle bugs cause a process that is not enough

Today, when checking the SMIDB, found in the CRS alarm log There are many errors, specifically:

650) this.width=650; "src="/e/u261/themes/default/images/spacer.gif "border=" 0 "style=" Background:url ("/e/u261/ Lang/zh-cn/images/localimage.png ") no-repeat center;border:1px solid #ddd;" alt= "Spacer.gif"/>2015-08-19 17:12:21.745:

[/oracle/app/11.2.0/grid_1/bin/oraagent.bin (6227)] crs-5013:agent  "/oracle/app/11.2.0/grid_1/bin/oraagent.bin"  failed to start process   "/oracle/app/11.2.0/grid_1/bin/lsnrctl"  for action  "Check": details at  "(: CLSN00008:) " in "/oracle/app/11.2.0/grid_1/log/smidb11/agent/crsd/oraagent_grid/oraagent_grid.log " 2015-08-19 17:13:09.986: [/oracle/app/11.2.0/grid_1/bin/oraagent.bin (6227)]CRS-5013:Agent  "/ Oracle/app/11.2.0/grid_1/bin/oraagent.bin " failed to start process "/oracle/app/ 11.2.0/grid_1/bin/lsnrctl " for action " Check ": details at " (: CLSN00008:) "  in  "/oracle/app/11.2.0/grid_1/log/smidb11/agent/crsd/oraagent_grid/oraagent_grid.log" 2015-08-19  17:13:21.758: [/oracle/app/11.2.0/grid_1/bin/oraagent.bin (6227)]crs-5013:agent  "/oracle/app/11.2.0/ Grid_1/bin/oraagent.bin " failed to start process "/ORACLE/APP/11.2.0/grid_1/bin/lsnrctl " for action " Check ": details at " (: CLSN00008:) " in   "/oracle/app/11.2.0/grid_1/log/smidb11/agent/crsd/oraagent_grid/oraagent_grid.log"

Further trace log discovery:

650) this.width=650; "src="/e/u261/themes/default/images/spacer.gif "border=" 0 "style=" Background:url ("/e/u261/ Lang/zh-cn/images/localimage.png ") no-repeat center;border:1px solid #ddd;" alt= "Spacer.gif"/>

2015-08-19 17:14:09.993: [ora. listener.lsnr][1342174976]{1:63186:26462} [check] clsn_agent::check: exception  sclsprocessspawnexception2015-08-19 17:14:21.744: [ora.asm][1342174976]{0:21:2} [check]  crscmd::clscrscmddata::stat entity 1 statflag 33 usefilter 02015-08-19  17:14:21.759: [ora.asm][1342174976]{0:21:2} [check] asmproxyagent::check clsagfw_res_ Status 02015-08-19 17:14:21.761: [ora. Listener_scan1.lsnr][1339545344]{0:21:2} [check] utils:execcmd action = 3 flags  = 38 ohome =  (NULL)  cmdname = lsnrctl. 2015-08-19  17:14:21.761: [ora. listener_scan1.lsnr][1339545344]{0:21:2} [check]  (: CLSN00008:) Utils:execcmd scls_process_spawn ( )  failed 12015-08-19 17:14:21.761: [ora. listener_scan1.lsnr][1339545344]{0:21:2} [check]  (: CLSN00008:)  categorY: -2, operation: fork, loc: spawnproc28, os error: 11, other:  forked failed [-1]2015-08-19 17:14:21.761: [ora. listener_scan1.lsnr][1339545344]{0:21:2} [check] clsnutils::error exception type=2  string=crs-5013: agent  "/oracle/app/11.2.0/grid_1/bin/oraagent.bin"  failed to start  process  "/oracle/app/11.2.0/grid_1/bin/lsnrctl"  for action  "check":  details  at  "(: CLSN00008:)"  in  "/oracle/app/11.2.0/grid_1/log/smidb11/agent/crsd/oraagent_grid/ Oraagent_grid.log "

Log of ONS:

650) this.width=650; "src="/e/u261/themes/default/images/spacer.gif "border=" 0 "style=" Background:url ("/e/u261/ Lang/zh-cn/images/localimage.png ") no-repeat center;border:1px solid #ddd;" alt= "Spacer.gif"/>

[Email protected] logs]$ tail ons.out pthread_create () Resource temporarily unavailablepthread_create () Resource Temporarily unavailablepthread_create () Resource temporarily unavailablepthread_create () Resource temporarily Unavailablepthread_create () Resource temporarily unavailablepthread_create () Resource temporarily unavailablepthread _create () Resource temporarily unavailablepthread_create () Resource temporarily unavailablepthread_create () Resource Temporarily unavailable[2015-05-07t03:09:22+08:00] [ONS] [trace:2] [] [internal] ONS worker process stopped (0)

This error indicates that the process failed to start due to insufficient system resources, check the Ulimit settings

650) this.width=650; "src="/e/u261/themes/default/images/spacer.gif "border=" 0 "style=" Background:url ("/e/u261/ Lang/zh-cn/images/localimage.png ") no-repeat center;border:1px solid #ddd;" alt= "Spacer.gif"/>

650) this.width=650; "src="/e/u261/themes/default/images/spacer.gif "border=" 0 "style=" Background:url ("/e/u261/ Lang/zh-cn/images/localimage.png ") no-repeat center;border:1px solid #ddd;" alt= "Spacer.gif"/>

[Email protected] logs]$ ulimit-u10240

Limit.conf

650) this.width=650; "src="/e/u261/themes/default/images/spacer.gif "border=" 0 "style=" Background:url ("/e/u261/ Lang/zh-cn/images/localimage.png ") no-repeat center;border:1px solid #ddd;" alt= "Spacer.gif"/>

# End of Filegrid soft nproc 10240grid hard nofile 65536oracle soft nproc 10240oracle hard Nofile 65536

There are some problems with limit.conf configuration, no hard nproc and soft nofle are configured, and fixed before restarting next Monday

650) this.width=650; "src="/e/u261/themes/default/images/spacer.gif "border=" 0 "style=" Background:url ("/e/u261/ Lang/zh-cn/images/localimage.png ") no-repeat center;border:1px solid #ddd;" alt= "Spacer.gif"/>

[[email protected] pam.d]$ cat login #%pam-1.0auth [user_unknown=ignore  success=ok ignore=ignore default=bad] pam_securetty.soauth        include      system-authaccount    required      pam_nologin.soaccount    include       system-authpassword   include      system-auth# pam_ selinux.so close should be the first session rulesession     required     pam_selinux.so closesession    required      pam_loginuid.sosession    optional      pam_console.so# pam_selinux.so open should only be followed by  Sessions to be executed in the user contextsession    required      pam_selinux.so opensession    required     pam_ Namespace.sosession    optional     pam_keyinit.so force  revokesession    include      system-auth-session    OPTIONAL     PAM_CK_CONNECTOR.SO[[EMAIL PROTECTED] PAM.D] $

The/etc/pam.d/login file does not have a resource restriction module added, and a row should be added here

Session required/lib64/security/pam_limits.so

After finding the information on the Internet, we found a document above the Oracle MOS, which is exactly the same as our situation:

The processes and resources started by CRS (Grid Infrastructure) does not inherit the Ulimit setting for "max User processes "From/etc/security/limits.conf Setting (document ID 1594606.1)

650) this.width=650; "src="/e/u261/themes/default/images/spacer.gif "border=" 0 "style=" Background:url ("/e/u261/ Lang/zh-cn/images/localimage.png ") no-repeat center;border:1px solid #ddd;" alt= "Spacer.gif"/>

Through verification, we found that although the ulimit-u of our grid users has been set to 10240, the actual operation is still 1024.

This is an Oracle bug 17301761 , our database version is 11.2.0.4, which is exactly the scope of the bug.

There are two solutions,

1. Patching

2. The method to evade by MoS is as follows:

The OHASD script needs to being modified to setthe Ulimit explicitly for all grids and database resources that is started BYT He Grid Infrastructure (GI).

1) go to Gi_home/bin

2) Make a backup of OHASD script file

        then
             $CLSECHO-phas-f crs-l-M 6021 "L" "unlimited"
        fi
         Ulimit-c Unlimited
        if ["$?"! = "0 "]
        then
             $CLSECHO-phas-f crs-l-M 6021 "C" "unlimited"
        fi
         ulimit-n 65536

In the above code, insert the following linejust before the line with "Ulimit-n 65536"

Ulimit-u 16384

4) Recycle CRS manually So, the Ohasdwill not with the new Ulimit setting for open files.
After the database was started, please issue "PS-EF | grep Pmon "Andget the PID of it.
Then, issue "cat/proc/<pid of the Pmon proces>/limits | Grepprocess "and find out if the Max process are set to 16384.
Setting the number of processes to 16384 should is enough for most serverssince have 16384 processes normally mean the S  Erver to Loaded veryheavily. Using smaller number like 4096 or 8192 should also suffice formost users.
In addition to above, the OHASD template needs to being modified to insure thatnew Ulimit setting persists even after a patch is applied.
1) go to Gi_home/crs/sbs

2) Make a backup of Crswrap.sh.sbs

3) in Crswrap.sh.sbs, insert the followingline just before the line "# Memlock limit are for Bug 9136459"

Ulimit-u 16384
Finally, although the above setting is successfully used to increase the NUMBEROF processes setting, please test this on T He test server first before Settingthe Ulimit on the production.

Reference: http://blog.csdn.net/weiwangsisoftstone/article/details/42460585

/limits.conf Oracle bugs cause a process that is not enough