In 11GR2 RAC environments, file system usage is tight and lsof shows a lot of ORAAGENT_ORACLE.L10 (deleted)
Reference text:
High Space Usage and "lsof" Output shows many ' Oraagent_oracle.l10 (deleted) ' in GI environment (DOC ID 1598252.1)
Suitable for:
Oracle database-enterprise edition-version 11.2.0.3 to 12.1.0.1 [Release 11.2 to 12.1]
Information in this document applies to any platform.
Symptoms:
Using the DF command, you can see that the GI home file system usage is quite high, but the du command does not see that the file is using all the space.
In this example, the DF display already uses 95% (about 89GB), but du shows only 34GB
The lsof command shows that many oraagent_oracle.l10 files are marked as deleted (---> Note: use lsof | grep deleted can locate files that have been deleted off), because handle has been holding these deleted files, so the OS cannot reclaim this part of the space.
Oracle 121508 Oracle 5w REG 253,3 10564931 6178664/u01/app/11.2.0.3/grid/log/dm01db01/agent/crsd/oraagent_oracle/ ORAAGENT_ORACLE.L10 (Deleted)
Oracle 121510 Oracle 5w REG 253,3 10564931 6178664/u01/app/11.2.0.3/grid/log/dm01db01/agent/crsd/oraagent_oracle/ ORAAGENT_ORACLE.L10 (Deleted)
Reason:
This is due to bug 17305100
While database is started by oraagent, the database processes inherit open file handles from the Oraagent and when the O RAAGENT_ORACLE.L10 is deleted as expected with the normal GI log file rotation policy, those handles is not closed by th E Database processes
Duplicates:bug 14062031 Bug 17034444
Solution:
The bug fix is included in 11.2.0.3 gi PSU9, 11.2.0.4 gi PSU1. For 12c, at the time of this writing, the issue is still being worked via bug 17387523
The workaround is to restart database or GI