cannot set user id: Resource temporarily unavailable

cannot set user id: Resource temporarily unavailable

前陣子，Infra報告無法透過putty以及SecureCRT串連到資料庫伺服器，提示的錯誤為Resource temporarily unavailable。由於該伺服器上有差不多有20個nstance，應該是超出了系統當前設定的值。關於超出資源限制的事之前有碰到過，只不過不是這個錯誤，而是open files: cannot modify limit: Operation not permitted。下面描述一下關於Resource temporarily unavailable這個錯誤及其相關資訊。

一、故障現象

installer@linux_02:~> ps -U Oracle |wc -l

2015

installer@linux_02:~> lsof | grep oracle|wc -l

83646

installer@linux_02:~> su - oracle

Password:

su: cannot set user id: Resource temporarily unavailable

#從Putty或者SecureCRT串連則會出現如下類似的錯誤：

Server sent disconnect message

type 2(protocol error):

"fork failed: Resource temporarily unavailable"

#系統日誌如下：

Aug 19 09:25:01 linux_02 /usr/sbin/cron[6736]: (oracle) MAIL (mailed 25 bytes of output butgot status 0x0001 )

Aug 19 09:26:29 linux_02 sshd[6864]: Accepted keyboard-interactive/pam for oracle from 192.168.9.1 port 1253 ssh2

Aug 19 09:26:29 linux_02 sshd[6875]: fatal: setresuid 2000: Resource temporarily unavailable

Aug 19 09:27:15 linux_02 sshd[6922]: Accepted keyboard-interactive/pam for oracle from 192.168.9.1 port 1312 ssh2

Aug 19 09:27:15 linux_02 sshd[6985]: fatal: setresuid 2000: Resource temporarily unavailable

二、故障分析

#以下是Metalink上788064.1對Resource temporarily unavailable的錯誤描述：

The error is different when it is reaching the limit 'open files' and 'max user processes' in /etc/profile .

A). Error on reaching the limit 'open files':

[oracle@mydesk~]$ssh rac2

oracle@rac2's password:

-bash: ulimit: max user processes: cannot modify limit: Operation not permitted

-bash: /home/oracle/.bash_profile: Too many open files

B). Error on reaching the limit 'max user processes':

[oracle@mydesk~]$ssh oracle@rac2

oracle@rac2's password:

-bash: ulimit: open files: cannot modify limit: Operation not permitted

-bash: fork: Resource temporarily unavailable

#也就是說我們當前的錯誤是由於進程數的限制而引起的 #Author : Leshami

#下面檢查當前limits.conf配置 #Blog : http://blog.csdn.net/leshami

linux_02:/etc/security # grep -v ^# /etc/security/limits.conf

oracle soft nproc 2047

oracle hard nproc 16384

oracle soft nofile 1024

oracle hard nofile 65536

#下面是來自SUSE官方的Resolution：

The affected user has reached the maximum number of process specified for him into the file /etc/security/limits.conf .

These commands, executed as root, can give you the number of process and of open files for the given user:

ps -U username | wc -l

lsof | grep username | wc -l

To fix the issue increase the NPROC Soft limit according to the user and applications needs; please look at the note below about the /etc/security/limits.conf file.

#上面的描述也是要增加nproc的值，

#從之前的運用的ps以及lsof命令來看，ps命令得到的oracle使用者的進程數為2015，從這個值來看，當前的配置符合要求。

#反而是開啟檔案數超出了設定的hard值，為83682。比較納悶的是不是檔案數錯誤，而是進程數錯誤。

#ID 788064.1對此給出的solution 如下：

a). Modify /etc/security/limits.conf manually

Increase the value of 'soft nofile' until it is equal to 'hard nofile' value. Increase the value of 'soft nproc' until it is equal to 'hard nproc' value.

#增加值到soft nofile的值到等於hard nofile的值；增加soft nproc的值到等於hard nproc的值。

#從這個來看的話，應該是系統一旦啟動後會分配soft的值，但是最大值不能超過hard值。不管系統是否空閑，都會分配soft值。

b). Install oracle-validated package to modify /etc/security/limits.conf #安裝oracle-validated來自動修改limits.conf(RHEL4 to OL5適用)

After install the oracle-validated package, the content of /etc/security/limits.conf is modified.

#Metalink ID 1239915.1上對此的描述也是要調整limits.conf

SOLUTION

The file /etc/security/limits.conf controls the resource limits for each user.

The nproc value determines how many processes that user is allowed and the nofile value limits the total number of files which may be opened at once.

Increase both the soft and hard limits for the destination user ("oracle" in our example) and save the file.

Activate The Changed User Limits

These changes take effect only upon login, so out and re-login to activate the changes.

三、故障解決

#根據上面的Solution調整limits.conf檔案

linux_02:/etc/security # cp limits.conf limits.conf.bk

linux_02:/etc/security # vi /etc/security/limits.conf

linux_02:/etc/security # grep -v ^# /etc/security/limits.conf

oracle soft nproc 16384

oracle hard nproc 16384

oracle soft nofile 65536

oracle hard nofile 65536

oracle@linux_02:/users/oracle> ps -U oracle |wc -l

2714

oracle@linux_02:/users/oracle> lsof | grep oracle|wc -l

110694

四、相關參考

ulimit: open files: cannot modify limit: Operation not permitted

DocID: 1239915.1

DocID: 788064.1