Configuring Hugepages for Oracle on Linux (x86-64)

• Introduction
• Configuring Hugepages
• Force Oracle to use Hugepages (use_large_pages)
• disabling Transparent hugepages (RHEL6/OL6)
• Configuring 1G Hugepagesize

Introduction

For large SGA sizes, hugepages can give substantial benefits in virtual memory management. Without hugepages, the memory of the SGA is divided into 4K pages, which has to be managed by the Linux kernel. Using hugepages, the page size is increased to 2MB (configurable to 1G if supported by the hardware), thereby reducing the Total number of pages to being managed by the kernel and therefore reducing the amount of memory required to the page T Able in memory. In addition to these changes, the memory associated with Hugepages can is swapped out, which forces the SGA to stay me Mory resident. The savings in memory and the effort of page management make hugepages pretty much mandatory for Oracle 11g Systems Runnin G on x86-64 architectures.

Just because you had a large SGA, it doesn ' t automatically mean you would have a problem if you don ' t use Hugepages. It is typically the combination of a large SGA and lots database connections, leads to problems. To determine how much memory is currently using to support the page table, run the following command at a time when T He server is under normal/heavy load.

# grep pagetables/proc/meminfopagetables:      1244880 kb#

Automatic Memory Management (AMM) is not compatible with Linux Hugepages, so apart from ASM instances and small Unimportan t databases, you'll probably has no need for AMM in a real database running on Linux. Instead, Automatic Shared Memory Management and Automatic PGA Management should be used as they is compatible with Hugepa Ges.

Configuring Hugepages

Run the following command to determine the current hugepage usage. The default hugepage size is 2MB to Oracle Linux 5.x and as you can see from the output below, by default No. hugepages is Defined.

$ grep huge/proc/meminfoanonhugepages:         0 kbhugepages_total:       0hugepages_free:        0HUGEPAGES_RSVD:        0hugepages_surp:        0Hugepagesize:       2048 kb$

Depending on the size of your SGA, your may wish to increase the value of hugepagesize to 1G.

Create a file called "hugepages_setting.sh" with the following contents.

#!/bin/bash## hugepages_settings.sh## Linux Bash script to compute values for the# recommended Hugepages/hugetlb Configura tion## note:this script does calculation for all GKFX memory# segments available when the script is run, no matter it# is a Oracle RDBMS shared memory segment or not.# Check for the kernel versionkern= ' uname-r | Awk-f. ' {printf ("%d.%d\n", $1,$2);} ' ' # Find out ' the Hugepage sizehpg_sz= ' grep hugepagesize/proc/meminfo | awk {' Print $ '} ' # Start from 1 pages to being on the safe side and guarantee 1 free hugepagenum_pg=1# cumulative number of P Ages required to handle the running GKFX memory segmentsfor seg_bytes in ' ipcs-m | awk {' Print $ '} | grep "[0-9][0-9]*" ' Do min_pg= ' echo ' $SEG _bytes/($HPG _sz*1024) "| Bc-q ' If [$MIN _pg-gt 0]; Then num_pg= ' echo ' $NUM _pg+ $MIN _pg+1 "| Bc-q ' fidone# Finish with resultscase $KERN in ' 2.4 ') hugetlb_pool= ' echo ' $NUM _pg* $HPG _sz/1024 "|          Bc-q '; echo "Recommended Setting:vm.hugetlb_pool = $HUGETLB _pool";;   ' 2.6 ' |    ' 3.8 ') echo "Recommended setting:vm.nr_hugepages = $NUM _pg";; *) echo "Unrecognized kernel version $KERN. Exiting. ";; Esac# End

Thanks to Bjoern Rost for pointing out of the issue when using the script against UEK3 and the suggested fix.

Make the file executable.

$ chmod u+x hugepages_setting.sh

Make sure all the Oracle services is running as normal on the server, then run the script and make a note of the Recommen Ded "Vm.nr_hugepages" value.

$./hugepages_setting.sh Recommended setting:vm.nr_hugepages = 305$

Edit the "/etc/sysctl.conf" file as the "root" user, adding the following entry, adjusted based on your output from the SC Ript. You should set the value greater than or equal to the value displayed by the script. You are only need 1 or 2 spare pages.

vm.nr_hugepages=306

One person reported also needing the hugetlb_shm_group setting on Oracle Linux 6.5. I did not and it was listed as a requirement for SUSE only. If you want to set it, get the ID of the DBA group.

# Fgrep dba/etc/groupdba:x:54322:oracle#

Use the resulting group ID in the "/etc/sysctl.conf" file.

vm.hugetlb_shm_group=54322

Run the following command as the "root" user.

# sysctl-p

Alternatively, edit the "/etc/grub.conf" file, adding "hugepages=306" to the end of the kernel line for the default kernel and reboot.

You can now see the hugepages has been created, but is currently not being used.

$ grep huge/proc/meminfoanonhugepages:         0 kbhugepages_total:     306hugepages_free:      306HUGEPAGES_RSVD:        0hugepages_surp:        0Hugepagesize:       2048 kb$

Add the following entries into the "/etc/security/limits.conf" script, where the setting are at least the size of the HUGEP Ages allocation in KB (Hugepages * hugepagesize). The value is 306*2048=626688.

* Soft Memlock 626688* hard Memlock 626688

If you prefer, you can set these parameters to a value just below the size of the physical memory of the server. This is the "can forget" about it, unless you add more physical memory.

Check The Memory_target parameters is not set for the database and Sga_target and Pga_aggregate_target parameters is Bei Ng used instead.

Sql> Show parameter TargetNAME                                 TYPE        VALUE------------------------------------------------------------- ----------------archive_lag_target                   integer     0db_flashback_retention_target        integer     1440fast_ Start_io_target                 integer     0fast_start_mttr_target               integer     0memory_max_target                    big Integer 0memory_target                        Big integer 0parallel_servers_target              integer     pga_aggregate_ Target                 Big integer 200msga_target                           big integer 600Msql>

Restart the server and Restart the database services as required.

Check the hugepages information again.

$ grep huge/proc/meminfoanonhugepages:         0 kbhugepages_total:     306hugepages_free:       98HUGEPAGES_RSVD:       93hugepages_surp:        0Hugepagesize:       2048 kb$

You can see the hugepages is now being used.

Remember, if you increase your memory allocation or add new instances, you need to retest the required number of hugepages , or risk Oracle running without them.

Force Oracle to use Hugepages (use_large_pages)

Sizing the number of hugepages correctly is important because prior to 11.2.0.3, if the whole SGA doesn ' t fit into the AVA Ilable Hugepages, the instance would start up without using any. From 11.2.0.3-onward, the SGA can run partly in hugepages and partly isn't, so the impact of this issue is great. Incorrect sizing May is obvious to spot. Later releases of the database display a "Large Pages Information" section in the alert log during startup.

Large pages Information *****************total Shared Global region in Large pages = 602 MB (100%) Large Pages used by this instance:301 (602 MB) Large pages unused system wide = 5 (Ten MB) (Alloc incr 4096 KB) Large pages confi gured System wide = 306 (612 MB) Large Page size = 2048 kb***********************************************************

If you is running Oracle 11.2.0.2 or later, you can set the Use_large_pages initialization parameter to ' only ' so the DAT Abase fails to start if it isn't backed by Hugepages. You can read more on this here.

ALTER SYSTEM SET use_large_pages=only scope=spfile; SHUTDOWN IMMEDIATE; STARTUP;

On startup The "Large page information" In the alert log reflects the use of this parameter.

Large Pages information *****************parameter use_large_pages = onlytotal Shared Global region in Large pages = 602 MB (100%) Large pages used by this instance:301 (602 MB) Large Pages unused system wide = 5 (ten MB) (all OC incr 4096 KB) Large Pages configured system wide = 306 (612 MB) Large Page size = 2048 kb******************************** ***************************

Attempting to start the database when there aren ' t enough hugepages to hold the SGA would now return the following error.

The "Large Pages Information" section of the alert log output describes the startup failure and the appropriate action to Take.

Large pages Information *****************parameter use_large_pages = onlylarge Pages Unused system wid E = 0 (0 KB) (alloc incr 4096 KB) Large Pages configured system wide = 0 (0 KB) Large Page size = 2048 kberror:  Failed t O Allocate shared global region with large pages, Unix errno =.  Aborting Instance startup.  Ora-27137:unable to allocate Large Pages to create a shared memory Segmentaction: Total  gkfx Global region size is 608 MB.  Increase the number of  unused large pages to atleast 304 (608 MB) to allocate 100% gkfx Global region with  large pages.***********************************************************

disabling Transparent hugepages (RHEL6/OL6)

Starting from Rhel6/ol6, Transparent Hugepages is implemented and enabled by default. They is meant to improve memory management by allowing Hugepages to being allocated dynamically by the "khugepaged" kernel t Hread, rather than at boot time like conventional hugepages. That's sounds like a good idea but unfortunately Transparent hugepages don ' t play well with Oracle databases and is Associ Ated with node reboots in RAC installations and performance problems on both single instance and RAC installations. As a result Oracle recommends disabling Transparent Hugepages on all servers running Oracle databases, as described in thi s MOS note.

• Alert:disable Transparent hugepages on SLES11, RHEL6, OEL6 and UEK2 kernels [ID 1557478.1]

You can check the current setting using the following command, which is displaying the default value of "Enabled=[always" .

# Cat/sys/kernel/mm/transparent_hugepage/enabled[always] Madvise never#

The preferred method to disable Transparent hugepages are to add "transparent_hugepage=never" to the kernel boot line in th E "/etc/grub.conf" file.

Title Oracle Linux Server (2.6.39-400.24.1.el6uek.x86_64)        root (hd0,0)        kernel/ vmlinuz-2.6.39-400.24.1.el6uek.x86_64 ro root=/dev/mapper/vg_ol6112-lv_root rd_no_luks  KEYBOARDTYPE=pc Keytable=uklang=en_us. UTF-8 rd_no_md sysfont=latarcyrheb-sun16  rd_no_dm rd_lvm_lv=vg_ol6112/lv_swap rd_lvm_lv=vg_ol6112/lv_root RHGB Quiet Numa=offtransparent_hugepage=never        initrd/initramfs-2.6.39-400.24.1.el6uek.x86_64.img

The server must is rebooted for this and take effect.

Alternatively, add the following lines into the "/etc/rc.local" File and reboot the server.

If Test-f/sys/kernel/mm/transparent_hugepage/enabled; Then   echo never >/sys/kernel/mm/transparent_hugepage/enabledfiif test-f/sys/kernel/mm/transparent_hugepage /defrag; Then   echo never >/sys/kernel/mm/transparent_hugepage/defragfi

Whichever method choose, remember to check the change have work after reboot.

# cat/sys/kernel/mm/transparent_hugepage/enabledalways Madvise [never]#

With Transparent hugepages disabled, you should proceed to configure conventional hugepages, as described above.

Configuring 1G Hugepagesize

As mentioned by Eugene in the comments, Oracle currently don ' t recommend using 1G hugepagesize. You can read more about the this in MOS Doc ID 1607545.1. With a, the rest of this section should probably is considered more than a academic exercise.

Check If your current hardware can support a hugepagesize of 1G. If The following command produces any output, it can.

# Cat/proc/cpuinfo | grep PDPE1GB

Thanks to Kevin Closson for pointing out of the hardware support requirement.

Edit the "/etc/grub.conf" file, adding the following entries on to the kernel line of the default grub entry. Adjust the "hugepages" entry to the desired number of 1G pages. Notice This includes the disabling of Transparent hugepages, which are not mandatory, but a good idea.

Transparent_hugepage=never hugepagesz=1g Hugepages=1 default_hugepagesz=1g

Check the current hugepages setup.

# grep Huge/proc/meminfohugepages_total:       0hugepages_free:        0HUGEPAGES_RSVD:        0hugepages_surp:        0Hugepagesize:       2048 kb#

Reboot and check the Hugepages setup again.

#  grep huge/proc/meminfohugepages_total:       1hugepages_free:        1HUGEPAGES_RSVD:        0hugepages_surp:        0Hugepagesize:    1048576 kb#

For more information see:

• Overview of Hugepages
• Configuring Linux hugepages for Oracle Database is Just Too difficult! Isn ' t It? Part–i.
• Huge pages and Transparent Huge pages
• Hugepages on Oracle Linux 64-bit [ID 361468.1]
• Hugepages on Linux:what It's ... And what It's not ... [ID 361323.1]
• Alert:disable Transparent hugepages on SLES11, RHEL6, OEL6 and UEK2 kernels [ID 1557478.1]
• Use_large_pages

Hope this helps. Regards Tim ...

Back to the Top.

Configuring Hugepages for Oracle on Linux (x86-64)