Linux配置hugepage，linuxhugepage

Linux配置hugepage，linuxhugepage

對於記憶體較大的Oracle資料庫伺服器，配置HugePage是很有必要的，下面是配置Huge的步驟：

我的環境：

OS：CentOS release 6.4 (Final)

Oracle：11.2.0.4 RAC
1.         設定memlock

編輯/etc/security/limits.conf

增加：

* Soft memlock 稍小於RAM值

* hard memlock 稍小於RAM值

Memlock值大於sga沒關係的，所以我們可以設定這個值在我們想要的SGA size和實體記憶體size之間，這個值得單位是kb。

[root@node1 ~]# grep -i memtot /proc/meminfo

MemTotal:       132250576 kB

我們實體記憶體大小126G，因此我可以設定memlock為100G。

在2個節點設定：

[root@node1 ~]# tail -2 /etc/security/limits.conf

* soft memlock 104857600

* hard memlock 104857600

設定完畢後，以Oracle使用者登入驗證：

node2-> ulimit -l

104857600

2.         禁用AMM特性：

禁用AMM特性需要將memory_max_target, memory_max_target2個參數重設，而不僅僅是設定為0，最好的辦法是建立一個pfile,在pfile中將這2個參數刪除，再根據這個pfile建立spfile.

主要記憶體參數設定為：

node1->

node1-> grep target /tmp/init.ora

*.pga_aggregate_target=1572864000

*.sga_target=64424509440

SQL> create spfile from pfile='/tmp/init.ora';

 

File created.

 

SQL> startup force

ORA-27102: out of memory

Linux-x86_64 Error: 28: No space left on device

Additional information: -134217728

Additional information: 1

SQL>

出現這個報錯是因為shmall設定的太小。

Shmall表示共用記憶體的總大小，單位為pagesize的大小。

根據我的配置，最大能支援8G的SGA：

node1-> getconf PAGE_SIZE

4096

node1->  cat /proc/sys/kernel/shmall

2097152

修改shmall並重新啟動資料庫，修改的方法很簡單，編輯/etc/sysctl.conf：

kernel.shmall= 16252928

即可。

修改完畢kernel.shmall，此時可以重啟資料庫。

SQL> startup nomount pfile='/tmp/init.ora';            

ORACLE instance started.

 

Total System Global Area 6.4137E+10 bytes

Fixed Size                  2269072 bytes

Variable Size            6979321968 bytes

Database Buffers         5.7043E+10 bytes

Redo Buffers              112762880 bytes

SQL>

SQL>

DB啟動後重新建立spfile：

SQL>  create spfile='+OCR/sszgdb/spfilesszgdb.ora' from pfile='/tmp/init.ora';

 

File created.

 

SQL>

3.         根據Oracle的指令碼計算hugepage大小（計算時要確保Oracle已經啟動），也可自己計算：

 

 

Press Enter to proceed...

 

Recommended setting: vm.nr_hugepages = 30739

node1->

4.         根據指令碼的計算值在2個節點編輯/etc/sysctl.conf

加入vm.nr_hugepages = 30739

[root@node2 ~]# grep -i huge /proc/meminfo

AnonHugePages:    602112 kB

HugePages_Total:   30739

HugePages_Free:    30739

HugePages_Rsvd:        0

HugePages_Surp:        0

Hugepagesize:       2048 kB停止資料庫執行個體

5.         重啟伺服器，一般重啟伺服器之後DB也自動啟動了。伺服器和DB重啟後可以通過以下命令驗證大頁的使用方式：

grep -i Hugepages /proc/meminfo

此時查看Oracle的alert日誌，也可以發現：

Starting ORACLE instance (normal)

************************ Large Pages Information *******************

Per process system memlock (soft) limit = UNLIMITED

 

Total Shared Global Region in Large Pages = 58 GB (96%)

 

Large Pages used by this instance: 29761 (58 GB)

Large Pages unused system wide = 24 (48 MB)

Large Pages configured system wide = 30739 (60 GB)

Large Page size = 2048 KB

 

RECOMMENDATION:

  Total System Global Area size is 60 GB. For optimal performance,

  prior to the next instance restart:

  1. Increase the number of unused large pages by

 at least 936 (page size 2048 KB, total size 1872 MB) system wide to

  get 100% of the System Global Area allocated with large pages

可以發現sga並沒有全部放在大頁裡，這樣可能對效能帶來負面影響，Oracle建議我們再加入936個大頁，可見Oracle的大頁尾本計算並不準確，我們需要增加大頁數量，並設定參數Use_large_pages=only，該參數確保Oracle的共用記憶體全部使用大頁。

再重啟DB，此時alert日誌：

************************ Large Pages Information *******************

Parameter use_large_pages = ONLY

Per process system memlock (soft) limit = 100 GB

 

Total Shared Global Region in Large Pages = 60 GB (100%)

 

Large Pages used by this instance: 30721 (60 GB)

Large Pages unused system wide = 0 (0 KB)

Large Pages configured system wide = 31675 (62 GB)

Large Page size = 2048 KB

可以發現sga全部使用了大頁了。

另需確保環境變數DISABLE_HUGETLBFS unset，該變數會禁用Oracle使用Hugepage

 

 

附：計算HugePage大小的指令碼：

#!/bin/bash

#

# hugepages_settings.sh

#

# Linux bash script to compute values for the

# recommended HugePages/HugeTLB configuration

#

# Note: This script does calculation for all shared memory

# segments available when the script is run, no matter it

# is an Oracle RDBMS shared memory segment or not.

#

# This script is provided by Doc ID 401749.1 from My Oracle Support

# http://support.oracle.com

 

# Welcome text

echo "

This script is provided by Doc ID 401749.1 from My Oracle Support

(http://support.oracle.com) where it is intended to compute values for

the recommended HugePages/HugeTLB configuration for the current shared

memory segments. Before proceeding with the execution please note following:

 * For ASM instance, it needs to configure ASMM instead of AMM.

 * The 'pga_aggregate_target' is outside the SGA and

   you should accommodate this while calculating SGA size.

 * In case you changes the DB SGA size,

   as the new SGA will not fit in the previous HugePages configuration,

   it had better disable the whole HugePages,

   start the DB with new SGA size and run the script again.

And make sure that:

 * Oracle Database instance(s) are up and running

 * Oracle Database 11g Automatic Memory Management (AMM) is not setup

   (See Doc ID 749851.1)

 * The shared memory segments can be listed by command:

     # ipcs -m

 

 

Press Enter to proceed..."

 

read

 

# Check for the kernel version

KERN=`uname -r | awk -F. '{ printf("%d.%d\n",$1,$2); }'`

 

# Find out the HugePage size

HPG_SZ=`grep Hugepagesize /proc/meminfo | awk '{print $2}'`

if [ -z "$HPG_SZ" ];then

    echo "The hugepages may not be supported in the system where the script is being executed."

    exit 1

fi

 

# Initialize the counter

NUM_PG=0

 

# Cumulative number of pages required to handle the running shared memory segments

for SEG_BYTES in `ipcs -m | cut -c44-300 | awk '{print $1}' | grep "[0-9][0-9]*"`

do

    MIN_PG=`echo "$SEG_BYTES/($HPG_SZ*1024)" | bc -q`

    if [ $MIN_PG -gt 0 ]; then

        NUM_PG=`echo "$NUM_PG+$MIN_PG+1" | bc -q`

    fi

done

 

RES_BYTES=`echo "$NUM_PG * $HPG_SZ * 1024" | bc -q`

 

# An SGA less than 100MB does not make sense

# Bail out if that is the case

if [ $RES_BYTES -lt 100000000 ]; then

    echo "***********"

    echo "** ERROR **"

    echo "***********"

    echo "Sorry! There are not enough total of shared memory segments allocated for

HugePages configuration. HugePages can only be used for shared memory segments

that you can list by command:

 

    # ipcs -m

 

of a size that can match an Oracle Database SGA. Please make sure that:

 * Oracle Database instance is up and running

 * Oracle Database 11g Automatic Memory Management (AMM) is not configured"

    exit 1

fi

 

# Finish with results

case $KERN in

    '2.2') echo "Kernel version $KERN is not supported. Exiting." ;;

    '2.4') HUGETLB_POOL=`echo "$NUM_PG*$HPG_SZ/1024" | bc -q`;

           echo "Recommended setting: vm.hugetlb_pool = $HUGETLB_POOL" ;;

    '2.6') echo "Recommended setting: vm.nr_hugepages = $NUM_PG" ;;

    '3.8') echo "Recommended setting: vm.nr_hugepages = $NUM_PG" ;;

esac

 

# End