A swap bug in the AIX6.1/11.2.0.3 database, aix6.111.2.0.3

A swap bug in the AIX6.1/11.2.0.3 database, aix6.111.2.0.3
Yesterday, I went to a customer in Nanjing to optimize the new business database. I checked alert. when a log is found in the past period of time, the following content is reported every several hours or intervals:
Thu Aug 21 09:01:26 2014 WARNING: Heavy swapping observed on system in last 5 mins. pct of memory swapped in [8.42%] pct of memory swapped out [2.16%]. please make sure there is no memory pressure and the SGA and PGAare configured correctly. look at DBRM trace file for more details.
Thu Aug 21 14:56:27 2014 WARNING: Heavy swapping observed on system in last 5 mins. pct of memory swapped in [5.40%] pct of memory swapped out [8.63%]. please make sure there is no memory pressure and the SGA and PGAare configured correctly. look at DBRM trace file for more details.
......
Sat Oct 18 22:13:48 2014 WARNING: Heavy swapping observed on system in last 5 mins. pct of memory swapped in [7.76%] pct of memory swapped out [0.33%]. please make sure there is no memory pressure and the SGA and PGAare configured correctly. look at DBRM trace file for more details.
The customer's environment is IBM P570, AIX 6.1, installed with Oracle 11.2.0.3 single-instance database, physical memory 64 GB, only 20 GB allocated to SGA, using memory Automatic Management
I checked out the MOS and found it was a bug on the AIX platform. The related documentation is: [1508575.1].
Corresponding databases and platforms:
Oracle Database-Enterprise Edition-Version 11.2.0.3 to 11.2.0.3 [Release 11.2] ibm aix on POWER Systems (64-bit)
Symptoms:

There is new warning message in alert. log in 11.2.0.3 similar

WARNING: Heavy swapping observed on system in last 5 mins.
Pct of memory swapped in [2.08%] pct of memory swapped out [0.12%].
Please make sure there is no memory pressure and the SGA and PGA
Are configured correctly. Look at DBRM trace file for more details.

On AIX platform this message can be seen even when there is no virtual memory swapping at all. -- physical memory is sufficient and swap space is not used at all

You may compare the vmstat from AIX level with DBRM trace file entries to see the differences.

Cause:

The issue is caused by unpublished Bug: 14731911.
 
Swap usage messages are based on statistics that do not reflect the actual usage.

The v $ osstat does not reflect proper stats for the swap space paging.

Solution:

Apply Patch: 11801934 on top of your ibm aix on POWER Systems (64-bit) platform.

P.S: Bug is port-specific. -- this bug is intended for the platform specified by the port.

The issue is fixed in patchset 11.2.0.4 and release 12.1. -- said it was fixed in patch 12.1, but in fact 12.1 still has this problem, there will be a ora-700 error, see The document: [ID 1919850.1]

Let's take a look at the BUG: 14731911 description:

Bug attributes

Type	B-Defect	Repaired in product version
Severity	2-Severe Loss of Service	Product Version	11.2.0.3
Status	96-Closed, Duplicate Bug	Platform	212-ibm aix on POWER Systems (64-bit)
Creation Time	2012-10-8	Platform version	6.1
Update Time	2014-10-11	Basic Bug	11801934
Database Version	11.2.0.3	Affected Platform	Port-Specific
Product source	Oracle	Knowledge related to this Bug, patches and bugs

Related Products

Product Line	Oracle Database Products	Series	Oracle Database Suite
Region	Oracle Database	Product	5-Oracle Database-Enterprise Edition

Hdr: 14731911 11.2.0.3 RDBMS 11.2.0.3 VOS PRODID-5 PORTID-212 11801934 Abstract: false swap warning messages printed to alert. log on aix *** 10/08/12 am *** bug type chosen ==================== Code SubComponent: virtual Operating System =============================================== = detailed problem description ========================== Oracle process seems to check wrong OS local statistic (which incl Ude also FILESYSTEM caching etc .) alert log shows WARNING: Heavy swapping observed on system in last 5 mins. pct of memory swapped in [2.08%] pct of memory swapped out [0.12%]. please make sure there is no memory pressure and the SGA and PGA are configured correctly. look at DBRM trace file for more details. but this is not reflected at OS level. diagnostic analysis ================================= 1. nmon sho Ws virtual memory swapping does not occur at all-see attached file -- nmon does not monitor swap Action 2. oracle Database Server is 11.2.0.3 and contains fix for 10220118 3. server configuration real mem: 144 GB lowest value of fre memory: GB-sufficient memory 4. DBRM seems to use a wrong OS statistics-trace file is attached WORKAROUND? ============ No technical impact ========================== Wrong diagnostic analyze. message is bothering customer's DBA when in fact the warning message is misleading related issues (bugs, forums, RFAs) ============================================ http://myforums.oracle.com/jive3/thread.jspa? ThreadID = 1104581 10220118 how often does the issue reproduce at customer site? ========================================================== ============== Always does the issue reproduce internally? ========================================================== No EXPLAIN WHY issue was not tested internally. ========================================================== ========= Unavailable Data Volume is a testcase available? ====================================== No Link to IPS Package: ================================ not available

DBRM (Database Resource Manager) is a background process that appears in the new feature of 11gR2. the log alarm log shows whether the OS has experienced intense swap activity in the last five minutes. On the AIX platform, due to the BUG: 14731911, this process of oracle falsely reports that the memory has performed swapin and swapout actions. We know that swap is used only when the physical memory is not enough (usually twice the physical memory ), swap is very performance-consuming (read and write from a physical disk ). However, I personally think this bug is not harmful, just in alert. a warning is reported in the log, which does not negatively affect the database. Therefore, whether to patch 11.2.0.4 is wise. If you want to enable alert. log is safe, so you can upgrade the patch. Of course, if the OS memory is insufficient, it is necessary to treat the swap action differently, because it does have a serious impact on the database. To identify whether the memory is insufficient, rather than system false positives, the analysis is mainly conducted through monitoring tools such as nmon, topas, and vmstat (free monitoring can also be used in linux)

For the AIX Platform, there is actually another bug, except for the unpublished base bug, rather than the port-specific bugAIX Platform.

If your Platform is IBM-AIX then this is not the only possible reason for this alert log message.
For ibm aix on POWER Systems (64-bit), there is also next known port-specific bug:
Bug 14731911-false swap warning messages printed to alert. LOG ON AIX
With unpublished base bug:
Bug 11801934: wrong page-in and page-out OS VM STATS IN AIX.

This WARNING information on the vmware platform, if not caused by a bug, is likely to be related to the ora-04031/ora-04030, which is much more serious

VMWare

Under VMWare, the messages may perhaps indicate a more serous issue, even when no memory related ORA-4031/ORA-4030 errors are reported.
Under circumstances, an instance in a virtual machine may be simply terminated by PMON due to error 471 without further errors in the alert log.
The OS logs may in such case report an out of memory condition like below:

[Root @ vmh ~] # Grep Kill/var/log/messages *
/Var/log/messages-20140629: Jun 27 18:29:06 vmh-msfc-dodp02 kernel: [1895074.304941] Out of memory: Kill process 42094 (oracle) score 391 or sacrifle ice child
/Var/log/messages-20140629: Jun 27 18:29:06 vmh-msfc-dodp02 kernel: [1895074.305203] Killed process 42094, UID 303, (oracle) total-vm: 189081588kB, anon-rss: 27412kB, file-rss: 109612

There are several solutions to solve the OS Memory swap problem:

1. diagnose whether a memory leakage process exists to solve the memory leakage. 2. optimize SGA/PGA to reduce oracle memory usage. use/proc/sys/vm/drop_caches to temporarily release some cache memory (Linux) 4. adjust the system VM memory management parameters, such as sysctl on Linux. the following parameters in conf:

Vm. min_free_kbytes: Raising the value in/proc/sys/vm/min_free_kbytes will cause the system to start reclaiming memory at an earlier time than it wowould have before.

Vm. vfs_cache_pressure: At the default value of vfs_cache_pressure = 100 the kernel will attempt to reclaim dentries and inodes at a "fair" rate with respect to pagecache and swapcache reclaim. decreasing vfs_cache_pressure causes the kernel to prefer to retain dentry and inode caches. increasing vfs_cache_pressure beyond 100 causes the kernel to prefer to reclaim dentries and inodes.

Vm. swappiness: Default 60, Apparently/proc/sys/vm/swappiness on Red Hat Linux allows the admin to tune how aggressively the kernel swaps out processes 'memory. decreasing the swappiness setting may result in improved Directory performance as the kernel holds more of the server process in memory longer before swapping it out.

Set the following values to reduce the possibility Of OOM (Out Of Memory:

# Oracle-Validated setting for vm. min_free_kbytes is 51200 to avoid OOM killer
VMS. min_free_kbytes = 51200
Vm. swappiness = 40
Vm. vfs_cache_pressure = 200

 
What is the difference between aix61 and aix53? After the upgrade, do applications need to be re-compiled and oracle databases need to be re-configured?

The version span is large. In theory, the application needs to be re-compiled and run and tested.
The Oracle database needs to query the official authentication information based on the specific version to see if it supports Aix 6.1. You may need to upgrade the Oracle database version or install the corresponding patch.

How to upgrade the system patch of AIX61, how to upload the patch to the AIX system, help me, and give a detailed step

1. Go to the IBM official website to download related OS Patches
2. Use ftp to upload files to the aix system, such as the tmp directory. You can use the command line method or the FTP tool.
3. Enter the cd directory.
4. Select update to latestxxxxxxx for smitty installp, and select (.) as the current directory.