Hugepages causes a sudden increase in system load due to the emergence of kswapd

On the linux server running the Oracle database, the load per minute in a certain period of time will suddenly rise to more than 40, and kswapd0 appears in the process queue, resulting in the database

On the linux server running the Oracle database, the load per minute in a certain period of time will suddenly rise to more than 40, and kswapd0 appears in the process queue, resulting in the database

On the linux server running the Oracle database, the load per minute in a certain period of time will suddenly rise to more than 40, and kswapd0 appears in the process queue, resulting in no response to the database, lasting for several minutes.
For applications, this time period is obviously stuck, just as the system has crashed.
If this occurs on a node in the Oracle RAC environment, the node may be restarted.
This is a very serious and fatal problem.

1. Problem
In this case, the database server memory is 96 GB, and the operating system linux RedHat 5 uses hugepages to manage part of the memory and runs a database instance. The database system version is 10.2.0.4.
The sga_max_size value of the database instance is 48 GB, And the pga_aggregate_target value is 24 GB.
Another instance is ASM, which is used to store database data files.
The result of top display during performance swing is as follows:

Top-13:21:11 up 49 days, 4 users, load average: 42.76, 14.42, 6.12
Tasks: 471 total, 26 running, 444 sleeping, 0 stopped, 1 zombie
Cpu (s): 87.2% us, 12.2% sy, 0.0% ni, 0.1% id, 0.1% wa, 0.1% hi, 0.2% si, 0.0% st
Mem: 98989932 k total, 98152840 k used, 837092 k free, 2056 k buffers
Swap: 50331636 k total, 12554316 k used, 37777320 k free, 37715224 k cached

Pid user pr ni virt res shr s % CPU % mem time + COMMAND
1057 root 20-5 0 0 R 100.2 0.0 991: 39.65 [kswapd0]

Zzz *** Fri Jun 29 13:21:35 CST 2012

Top-13:21:39 up 49 days, 4 users, load average: 28.99, 13.85, 6.19
Task: 471 total, 2 running, 468 sleeping, 0 stopped, 1 zombie
Cpu (s): 0.2% us, 8.4% sy, 0.0% ni, 91.3% id, 0.1% wa, 0.0% hi, 0.0% si, 0.0% st
Mem: 98989932 k total, 98104680 k used, 885252 k free, 3028 k buffers
Swap: 50331636 k total, 12801004 k used, 37530632 k free, 37606456 k cached

Pid user pr ni virt res shr s % CPU % mem time + COMMAND
1057 root 20-5 0 0 R 100.3 0.0 992: 07.44 [kswapd0]
12530 root 15 0 13032 1400 R 812 0.7. 03 top-B-c-n 2
1376 root 10-5 0 0 S 0.3 0.0. 23 [scsi_eh_1]

It can be seen that the load on the server has increased sharply in one minute, but it has been up to 42, and it is only about 4 if it is normal. In the running process queue, kswapd0 is displayed.
The operating system will wake up kswapd every time and check whether the memory is tight. If not, sleep. In kswapd, there are two thresholds: pages_hige and pages_low, when the number of idle memory pages is lower than pages_low, The kswapd process will scan the memory and release 32 free pages each time until the number of free pages reaches pages_high.

By checking the output result of vmstat, it is found that during that period, the system's page switching is very serious.
Zzz *** Fri Jun 29 13:22:05 CST 2012
Procs ----------- memory ---------- --- swap -- ----- io ---- system -- ----- cpu ------
R B swpd free buff cache si so bi bo in cs us sy id wa st
7 0 13022188 919216 3064 37429916 27 17 203 0 0 3 1 95 1 0
4 0 13040224 914712 3116 37411924 680 2196 1450 2844 1387 1654 13 10 77 0
2 0 13058536 919060 3216 37395064 104 1444 1118 1492 1235 16 9 75 0
Zzz *** Fri Jun 29 13:22:35 CST 2012
Procs ----------- memory ---------- --- swap -- ----- io ---- system -- ----- cpu ------
R B swpd free buff cache si so bi bo in cs us sy id wa st
9 0 13573224 912300 3356 36885376 27 17 203 0 0 3 1 95 1 0
8 0 13573088 913408 3368 36885224 276 3740 5643 1643 7707 31 9 60 0
1 0 13572892 913300 3368 36885920 252 374 3955 2209 9632 14 9 77 0

That is to say, the problem is that the memory is insufficient, resulting in frequent use of swap partitions. The kswapd0 process needs to swap in and out the virtual memory disk space, which causes the system to swing for a short time.
25576 pages in the physical memory of the operating system are not used. Therefore, the kswap process will certainly perform the Virtual Memory page change operation.

2. Analysis
Since the problem occurs in memory usage, but the physical memory configuration of our server is 96 GB, the memory configuration of the database instance is also reasonable.
However, there is indeed a page change in the SWAp partition in vmstat.
What we need to analyze is how the 96 GB memory is used?
In the environment introduction, we introduced how to manage memory in the large memory management mode. Therefore, we first query the memory information about large memory usage.
[Root @ db3 ~] # Cat/proc/meminfo | grep-I huge
HugePages_Total: 25576
HugePages_Free: 25517
HugePages_Rsvd: 4
Hugepagesize: 2048 kB

Wow, I was surprised when I saw the result. The large memory has 25517 pages not used. The size of a page is 2 MB. That is to say, the 51034 MB memory is limited by the large memory management mechanism and is idle, other processes on the system cannot be used. Only 118 MB of memory is available for 59 pages.
(Note: Why does the new process use the virtual memory space instead of the memory area? This is a question. It may be because the SGA of the new database instance has used the remaining 48 memory, which is not enough, so virtual memory is used .)

In the system configuration, the size of hugepages is set to 25576, which is 48 GB memory, half of the physical memory.
[Root @ db3 ~] # Sysctl-p
Vm. nr_hugepages = 25576

Use ipcs-m to view the shared memory segments used by oracle users as follows:
------ Shared Memory Segments --------
Key shmid owner perms bytes nattch status
0xb0af65c0 3047451 oracle 600 132120576 13
0x00007bd98 3702814 oracle 640 51541704704 132
The maximum value is 51541704704 bytes, Which is 49154 MB. All 'shared _ pool_size ', 'large _ pool_size', 'java _ pool_size ', 'streams _ pool_size', and 'db _ cache_size 'in the SGA.
Hugepages is not used at all for the shared memory segments of Oracle users.

Then, check the system log and find that an oracle instance used the hugepages, and the current instance was started later, so another memory is used.
Later, the previous instance was closed, but the memory space in hugepages had been idle, and the new instance could not use the memory space.

3. Solution
The problem has been analyzed here, but there are actually solutions.
After restarting the database instance, we can use the hugepages memory space.
If the instance cannot be restarted, we can also downgrade the settings by setting the value of nr_hugepages. However, this is only a personal suggestion and does not rule out the emergence of new problems.

4. Technology hugepages
To increase the size of hugepages, restart the operating system.
If you think this is a memory parameter value, you can use sysctl-w to modify it. This is incorrect. In terms of memory management, hugepages requires a large number of consecutive memory areas. Otherwise, serious memory fragments will affect the system performance.

The hugepages Technology of linux has gradually become popular with large memory servers in recent years. As a result, hugepages has also become a problem.
This article analyzes and summarizes the problems encountered in hugepages.