Use HugePages to optimize memory performance

Use HugePages to optimize memory performance

 

1. Introduction

The system process accesses the memory through a virtual address, but the CPU must convert it to the physical memory address to truly access the memory. To improve the conversion efficiency, the CPU caches the ing between the nearest virtual memory address and the physical memory address and stores it in a CPU-maintained ing table. To increase the memory access speed as much as possible, you need to save as many mappings as possible in the ing table.

In Linux, memory is divided into pages. By default, each page is 4 kb, which means that if the physical memory is large, there will be a lot of entries in the ing table, which will affect the retrieval efficiency of the CPU. Because the memory size is fixed, to reduce the number of ing table entries, you can only increase the page size.

2. HugePages Introduction

 

2.1 concepts

HugePages was introduced in Linux2.6 kernel, mainly providing 4 K page and relatively large page selection.

 

Concept	Concepts
Page table	Page table is the data structure model of the virtual memory system on the operating system. It is used to store the correspondence between virtual addresses and physical addresses. When we access the memory, we first access the page table, and then Linux accesses the real physical memory (ram + swap) through the page table mapping)
TLB	A Translation Lookaside Buffer (TLB) TLB is a fixed-size buffer (or cache) allocated in the cpu. It is used to save part of the page table content, so that the CPU can access it more quickly and perform address conversion.
Hugetlb	Hugetlb is an entry recorded in TLB and points to Hugepages.
Hugetlbfs	This is a new memory file system based on 2.6 kernel, like tmpfs. In TLB, hugetlb is used to point to hugepage. The allocated hugepage is provided to the process as the memory file system hugetlbfs (similar to tmpfs.

2.2 significance of using HugePages

HugePages is a feature of the Linux kernel. hugepage can replace the traditional 4 K page with a larger memory page. HugePage offers the following benefits:

1. HugePages will be allocated directly when the system is started and the corresponding memory area will be retained.

2. HugePages will not be released or changed after it is started without administrator intervention.

3. No swap.

Notswappable: HugePages are not swappable. Therefore thereis no page-in/page-outmechanic overhead. HugePages are universally regarded aspinned.

4. The memory size of the page table stored in the CPU cache is greatly increased, thus increasing the TLB hit rate.

The virtual memory address segment of the process first connects to the page table and then to the physical memory. Therefore, when accessing the memory, you must first access the page tables to obtain the ing between the virtual memory and the physical memory, and then access the physical memory.

Some TLB In the CPU cache is used to store some page tables to increase the conversion speed. Because the page size increases, the size of the TLB with the same size also increases. This improves the TLB hit rate and address translation speed.

5. Reduce the load on the page table.

If HugePages is not used during the XXX system performance test, the pagetable size on the database server is about 5 GB (this should be the main cause of insufficient memory on the database server during the performance test ):

Node74:/home/oracle # cat/proc/meminfo

MemTotal: 16323732 kB

PageTables: 5442384kB

After HugePages is configured, The pagetable size is only 124 MB (the memory usage is stable at around 80% during performance testing ):

Node74:/home/oracle # cat/proc/meminfo

MemTotal: 16323732 kB

PageTables: 127384 kB

Eliminated page tablelookup overhead:Because hugepage is not swappable, there is no page table lookups.

Faster overall memory performance:Because the virtual memory requires two steps to actually correspond to the physical memory address, less pages reduces page table access and avoids page table hotspot bottlenecks.

6. improve memory performance and reduce CPU load. The principle is the same as above.

2.3 precautions for using HugePages

1. Hugepages will be reserved after allocation. The size of Hugepages must be larger than the sum of the SGA instances on the server.

For example, if Hugepages is set to 8 GB and oracle SGA is set to 9 GB, then oracle will not use the 8 GB Hugepages at startup. This 8 GB is a waste. Therefore, the size of the SGA must be calculated when Hugepages is set. A script is provided later.

2. Other processes cannot use Hugepages memory, so do not set it to too large. It is a little bigger than SGA to ensure that SGA can use hugepages.

3. There are four items related to Hugepage in meminfo:

HugePages_Total: 4611

HugePages_Free: 474

HugePages_Rsvd: 467

Hugepagesize: 2048 kB

HugePages_TotalReturns the size of the allocated memory by multiplying the number of pages and Hugepagesize. 4611*2/1024 about 9 GB

HugePages_FreeThe number of Hugepages that have never been used. Even though oraclesga has allocated this part of memory, if there is no actual write, we can still see it as Free. This is easy to misunderstand.

HugePages_RsvdNumber of pages that have been reserved but not used. When Oracle was just started, most of the memory should be Reserved and Free. With the use of oracle SGA, Reserved and Free will continue to decrease.

HugePages_Free-HugePages_RsvdThis part is not used memory. If there is no other oracle instance, this part of memory may never be used, that is, wasted.

4. HugePages and oracle AMM (Automatic Memory Management) are mutually exclusive. Therefore, you must set the memory parameter MEMORY_TARGET/MEMORY_MAX_TARGET to 0 when using HugePages.

3. Configure HugePages 3.1 modify the Kernel Parameter memlock

Modify the Kernel Parameter memlock in KB. If the memory is 16 GB, the size of memlock is slightly smaller than the physical memory. We plan to lock the memory size of 12 GB. It is no harm to set the parameter to be greater than SGA.

Log on to the two database servers as the root user and edit the limits. conf file:

Node74 :~ # Vi/etc/security/limits. conf

Add the following two lines:

* Soft memlock 12582912

* Hard memlock 12582912

3.2 verify memlock limit

Log on to the root and oracle users again and check memlocklimit.

Node74 :~ # Ulimit-l

12582912

Oracle @ node74: ~> Ulimit-l

12582912

3.3 disable AMM

If you use a version of 11G or later, AMM is enabled by default, but AMM and Hugepages are incompatible, you must disable AMM first. To disable AMM, follow these steps:

3.3.1 shut down database instances

Oracle users have logged on to two database servers and used sqlplus to close two database instances.

Oracle @ node74: ~> Sqlplus/as sysdba

SQL> shutdown immediate

3.3.2 create a pfile

Log on to one of the hosts as an oracle user and run the following command to create a pfile:

Oracle @ node74: ~> Sqlplus/as sysdba

SQL> create pfile = '/home/oracle/pfile. ora' fromspfile = '+ DG_ORA/orcl/spfileorcl. ora ';

3.3.3 edit pfile

Edit pfile and delete the memory_max_target and memory_target parameters:

Oracle @ node74: ~> Vi/home/oracle/pfile. ora

Delete the following lines:

Orcl1.memory _ max_target = 11114905600

Orcl2.memory _ max_target = 11114905600

*. Memory_max_target = 0

Orcl1.memory _ target = 11114905600

Orcl2.memory _ target = 11114905600

*. Memory_target = 0

Save the modified file.

3.3.4 create a spfile

Run the following command to create a spfile:

Oracle @ node74: ~> Sqlplus/as sysdba

SQL> create spfile = '+ DG_ORA/orcl/spfileorcl. ora' from pfile = '/home/oracle/pfile. ora ';

3.3.5 modify system parameter kernel. shmall

Kernel. shmall is the maximum size of shared memory that can be used by the system at a time. The Unit is page (4 kb ). After you disable AMM, You need to modify the system parameter kernel. shmall. If this parameter is set too small, it may cause a database startup failure ORA-27102 (see Appendix 4.2 ).

ORACLE recommends setting it to the sum of the SGA of all database instances in the system. For example, if the sum of SGA is 9 GB, set kernel. shmall = 9*1024*1024/4 = 2359296.

Log on to the two database servers as the root user and edit the sysctl. conf file.

Node74 :~ # Vi/etc/sysctl. conf

Modify the kernel. shmall parameter:

Kernel. shmall = 2359296

Run sysctl-p to make the configuration take effect:

Node74 :~ # Sysctl-p

3.3.6 start a database instance

Log on to two database servers as an oracle user and start two database instances through sqlplus.

Oracle @ node74: ~> Sqlplus/as sysdba

SQL> startup

3.4 required for computing Hugepage page size

Make sure all instances have been started (including ASM) and run hugepages_settings.sh as the root user (for the script content, see appendix 4.1) to evaluate the size of Hugepages to be set.

Node74:/home/oracle #./hugepages_settings.sh

This script is provided by Doc ID 401749.1 from MyOracle Support

(Http://support.oracle.com) where it is intended tocompute values

The recommended HugePages/HugeTLB configuration forthe current shared

Memory segments. Before proceeding with the executionplease make sure

That:

* OracleDatabase instance (s) are up and running

* OracleDatabase 11g Automatic Memory Management (AMM) is not setup

(See Doc ID749851.1)

* The sharedmemory segments can be listed by command:

# Ipcs-m

Press Enter toproceed...

---- Press Enter directly

Recommended setting: vm. nr_hugepages =4611

It can also be calculated manually:

Nr_hugepages> = SGA_Target/Hugepagesize

= 9 GB * 1024 M/2 M

= 4608

Take a value slightly greater than 4608.

3.5 modify vm. nr_hugepages Parameters

Log on to the two database servers as the root user and edit/etc/sysctl. conf:

Node74 :~ # Vi/etc/sysctl. conf

Modify the vm. nr_hugepages parameter to the value calculated in the previous step:

Vm. nr_hugepages = 4611

Run sysctl-p to make the configuration take effect:

Node74 :~ # Sysctl-p

3.6 stop the database instance and restart the Operating System

Stop all database instances and restart the operating system. (In theory, the operating system does not need to be restarted. We recommend that you restart the operating system)

3.7 check whether the settings take effect

After the system restarts, start all the databases and run the following command to check whether the configuration takes effect:

Node74 :~ # Grep HugePages/proc/meminfo

HugePages_Total: 4611

HugePages_Free: 2394

HugePages_Rsvd: 2387

HugePages_Surp: 0

HugePages_Free <HugePages_Total indicates that Hugepages have taken effect, and HugePages_Rsvd is not 0.

4. Appendix
4.1 script hugepages_settings.sh

#! /Bin/bash   #   # Hugepages_settings.sh   #   # Linux bash script to compute values for   # Recommended HugePages/HugeTLB configuration   #   # Note: This script does calculation for all shared memory   # Segments available when the script is run, no matter it   # Is an Oracle RDBMS shared memory segment or not.   #   # This script is provided by Doc ID 401749.1 from My Oracle Support   # Http://support.oracle.com       # Welcome text   Echo"   This script is provided by Doc ID 401749.1 from My Oracle Support   (Http://support.oracle.com) where it is intended to compute values   The recommended HugePages/HugeTLB configuration for the current shared   Memory segments. Before proceeding with the execution please make sure   That:   * Oracle Database instance (s) are up and running   * Oracle Database 11g Automatic Memory Management (AMM) is not setup   (See Doc ID 749851.1)   * The shared memory segments can be listed by command:   # Ipcs-m   Press Enter to proceed ..."   Read   # Check for the kernel version   KERN = 'uname-r | awk-F. '{printf ("% d. % d \ n", $1, $2 );}''   # Find out the HugePage size   HPG_SZ = 'grep Hugepagesize/proc/meminfo | awk '{print $2 }''   # Initialize the counter   NUM_PG = 0   # Cumulative number of pages required to handle the running shared memory segments   For SEG_BYTES in 'ipcs-m | awk' {print $5} '| grep "[0-9] [0-9] *"'   Do   MIN_PG = 'echo "$ SEG_BYTES/($ HPG_SZ * 1024)" | bc-Q'   If [$ MIN_PG-gt 0]; then   NUM_PG = 'echo "$ NUM_PG + $ MIN_PG + 1" | bc-Q'   Fi   Done   RES_BYTES = 'echo "$ NUM_PG * $ HPG_SZ * 1024" | bc-Q'   # An SGA less than 100 MB does not make sense   # Bail out if that is the case   If [$ RES_BYTES-less than 100000000]; then   Echo "***********"   Echo "** ERROR **"   Echo "***********"   Echo "Sorry! There are not enough total of shared memory segments allocated   HugePages configuration. HugePages can only be used for shared memory segments   That you can list by command:   # Ipcs-m   Of a size that can match an Oracle Database SGA. Please make sure that:   * Oracle Database instance is up and running   * Oracle Database 11g Automatic Memory Management (AMM) is not configured"   Exit 1   Fi   # Finish with results   Case $ KERN in   '2. 4') HUGETLB_POOL = 'echo "$ NUM_PG * $ HPG_SZ/1024" | bc-Q ';   Echo "Recommended setting: vm. hugetlb_pool = $ HUGETLB_POOL ";;   '2. 6') echo "Recommended setting: vm. nr_hugepages = $ NUM_PG ";;   *) Echo "Unrecognized kernel version $ KERN. Exiting .";;   Esac   # End

4.2 database startup error ORA-27102

Upon startup of Linux database getORA-27102: out of memory Linux-X86_64 Error: 28: No space left on device

Subject:

Upon startup of Linux database getORA-27102: out of memory Linux-X86_64 Error: 28: No space left on device

Doc ID:

Note: 301830.1

Type:

PROBLEM

Last Revision Date:

31---2008

Status:

PUBLISHED

In this Document

Symptoms

Changes

Cause

Solution

References

--------------------------------------------------------------------------------

Applies:

OracleServer-Enterprise Edition-Version: 9.2.0.4 to 11.1.0.6

UnitedLinux x86-64

Red Hat Enterprise Linux Advanced Serverx86-64 (AMD Opetron Architecture)

X86 64 bit

Symptoms

When trying to increase the SGA to approachhalf available RAM with an Oracle 64bit version on a Linux 64bit operatingsystem, even though shmmax is set to match half the amount of RAM, youget the following error when trying to start the instance:

SQL> startup nomount

ORA-27102: out of memory

Linux-x86_64 Error: 28: No space left ondevice

Changes

Shmall is too small, most likely is set tothe default setting of 2097152

$ Cat/proc/sys/kernel/shmall

2097152

Cause

Shmall is the total amount of sharedmemory, in pages, that the system can use at one time.

Solution

Set shmall equal to the sum of all the SGAson the system, divided by the page size.

The page size can be determined using thefollowing command:

$ Getconf PAGE_SIZE

4096

For example, if the sum of all the SGAs onthe system is 16 Gb and the result of '$ getconf PAGE_SIZE' is 4096 (4 kb) then set shmall to 4194304 (4 Mb)

As the root user set the shmall to 419100004in the/etc/sysctl. conf file:

Kernel. shmall = 4194304

Then run the following command:

# Sysctl-p

# Cat/proc/sys/kernel/shmall

4194304

NOTE:

The above command loads the new value and areboot is not necessary

Switch back to being the oracle user andretry the startup command.

References

Note 169706.1-Oracle? Database onAIX ?, HP-UX ?, Linux ?, Mac OS? X, Solaris ?, Tru64 Unix? Operating SystemsInstallation and Configuration Requirements Quick Reference (8.0.5 to 11.1)

Keywords

NO ~ SPACE ~ LEFT ~ ON ~ DEVICE; START ~ INSTANCE; OUT ~ OF ~ MEMORY; 64BIT;