HBase parameter Configuration and description

Transferred from: http://www.cnblogs.com/nexiyi/p/hbase_config_94.html

The goal is to look at the configuration of the production environment and the default configuration.

Hbase.hregion.max.filesize:100ghbase.regionserver.hlog.blocksize:512mhbase.regionserver.maxlogs:32

.............

Version: 0.94-cdh4.2.1hbase-site.xml configuration Hbase.tmp.dir

• The local file system TMP directory, generally configured as local mode settings, but it is best to set up, because many files will be set by default to the following
• On-line configuration

  <property>     <name>hbase.tmp.dir</name>     <value>/mnt/dfs/11/hbase/hbase-tmp</value></property>

• Default value:

  ${java.io.tmpdir}/hbase-${user.name}

  Write to the system's/tmp directory

Hbase.rootdir

• All Regionserver shared Directories in the HBase cluster, used to persist hbase data, are typically set up in HDFs file directories, such as Hdfs://namenode.example.org:9000/hbase
• On-line configuration

  <property>     <name>hbase.rootdir</name>     <value>hdfs://mycluster/hbase</value></property>

• Default value:

  ${hbase.tmp.dir}/hbase

hbase.cluster.distributed

• Cluster mode, distributed or standalone mode, if set to False, the hbase process and the zookeeper process are in the same JVM process.
• True on-line configuration
• Default value: False

Hbase.zookeeper.quorum

• Zookeeper the URL configuration of the cluster, separated by commas (,) between multiple hosts
• On-line configuration

  <property>   <name>hbase.zookeeper.quorum</name>     <value>inspurXXX.xxx.xxx.org,inspurXXX.xxx.xxx.org,inspurXXX.xxx.xxx.org,inspurXXX.xxx.xxx.org,inspurXXX.xxx.xxx.org</value></property>

• Default value: localhost

Hbase.zookeeper.property.dataDir

• The configuration in the Zookeeper zoo.conf. Where the snapshot is stored
• On-line configuration:/home/hadoop/zookeeperdata
• Default value: ${hbase.tmp.dir}/zookeeper

Zookeeper.session.timeout

• Client-to-ZK connection time-out
• On-line configuration: 1200000 (20min)
• Default value: 180000 (3min)

Hbase.zookeeper.property.tickTime

• The time interval between client side and ZK sending heartbeat
• On-line configuration: 6000 (6s)
• Default value: 6000

Hbase.security.authentication

• HBase cluster security authentication mechanism, the current version only supports Kerberos security authentication.
• On-line configuration: Kerberos
• Default value: Empty

Hbase.security.authorization

• Whether HBase opens a security authorization mechanism
• On-line configuration: True
• Default value: False

Hbase.regionserver.kerberos.principal

• Regionserver principal name for Kerberos authentication (consists of three parts: service or user name, instance name, and domain name)
• On-line configuration: Hbase/[email protected]xxx.xxx.com
• Default: None

Hbase.regionserver.keytab.file

• Regionserver keytab file path
• On-line configuration:/home/hadoop/etc/conf/hbase.keytab
• Default value: None

Hbase.master.kerberos.principal

• Principal name of Kerberos authentication for master (consisting of three parts: service or user name, instance name, and domain name)
• On-line configuration: Hbase/[email protected]xxx.xxx.com
• Default: None

Hbase.master.keytab.file

• Master keytab file path
• On-line configuration:/home/hadoop/etc/conf/hbase.keytab
• Default value: None

Hbase.regionserver.handler.count

• Regionserver the number of threads processing IO requests
• On-line configuration: 50
• Default configuration: 10

Hbase.regionserver.global.memstore.upperLimit

• Regionserver process block Flush trigger condition: Memstore of all region on the node reaches upperlimit*heapsize
• On-line configuration: 0.45
• Default configuration: 0.4

Hbase.regionserver.global.memstore.lowerLimit

• A condition for the regionserver process to trigger flush: the sum of all region Memstore on the node reaches lowerlimit*heapsize
• On-line configuration: 0.4
• Default configuration: 0.35

Hbase.client.write.buffer

• Client writes buffer, setting AutoFlush to False when client writes full buffer to flush
• On-line configuration: 8388608 (8M)
• Default configuration: 2097152 (2M)

Hbase.hregion.max.filesize

• The region size of a single columnfamily, and if you follow the Constantsizeregionsplitpolicy policy, the value set above is automatically split
• On-line configuration: 107374182400 (100G)
• Default configuration: 21474836480 (20G)

Hbase.hregion.memstore.block.multiplier

• More than multiples of memstore size reach this value then block all write requests, self-protection
• On-line configuration: 8 (the memory is large enough to be properly resized, this situation requires the client to make adjustments)
• Default configuration: 2

Hbase.hregion.memstore.flush.size

• Memstore size, when this value is reached, it is flush to the external memory device
• On-line configuration: 104857600 (100M)
• Default value: 134217728 (128M)

Hbase.hregion.memstore.mslab.enabled

• Whether to turn on the Mslab scheme to reduce the full GC due to memory fragmentation and improve overall performance
• On-line configuration: True
• Default configuration: True

Hbase.regionserver.maxlogs

• Regionserver Number of Hlog
• On-line configuration: 128
• Default configuration: 32

Hbase.regionserver.hlog.blocksize

• Hlog size limit, reach this value block, roll off
• On-line configuration: 536870912 (512M)
• Default configuration: Block size for HDFS configuration

Hbase.hstore.compaction.min

• Minimum number of storefiles to enter the minor compact queue
• On-line configuration: 10
• Default configuration: 3

Hbase.hstore.compaction.max

• The maximum number of files in a single minor compact
• On-line configuration: 30
• Default configuration: 10

Hbase.hstore.blockingStoreFiles

• When the number of storefile in a region reaches this value, Block writes, waiting for the compact
• On-line configuration: 100 (production environment can be set very large)
• Default configuration: 7

Hbase.hstore.blockingWaitTime

• Block's wait Time
• On-line configuration: 90000 (90s)
• Default configuration: 90000 (90s)

Hbase.hregion.majorcompaction

• Cycles that trigger the major compact
• On-line configuration: 0 (switch off the major compact)
• Default configuration: 86400000 (1d)

Hbase.regionserver.thread.compaction.large

• Large the number of threads in the compact thread pool
• On-line configuration: 5
• Default configuration: 1

Hbase.regionserver.thread.compaction.small

• Small the number of threads in the compact thread pool
• On-line configuration: 5
• Default configuration: 1

Hbase.regionserver.thread.compaction.throttle

• Compact (major and minor) requests to enter the critical point of the large and small compact thread pool
• On-line configuration: 10737418240 (10G)
• Default configuration: 2 * this.minfilestocompact * this.region.memstoreFlushSize

Hbase.hstore.compaction.max.size

• Minor Compact queue storefile file Max size
• On-line configuration: 21474836480 (20G)
• Default configuration: Long.max_value

Hbase.rpc.timeout

• RPC Request Timeout time
• On-line configuration: 300000 (5min)
• Default configuration: 60000 (10s)

Hbase.regionserver.region.split.policy

• Split Operation Default Policy
• On-line configuration: Org.apache.hadoop.hbase.regionserver.ConstantSizeRegionSplitPolicy (take the old strategy, own control split)
• Default configuration: Org.apache.hadoop.hbase.regionserver.IncreasingToUpperBoundRegionSplitPolicy (if the region does not reach MaxFileSize, if FileSize reaches Regio Ncount * Regioncount * flushsize split operation)

Hbase.regionserver.regionSplitLimit

• Maximum number of region on a single regionserver
• On-line configuration: 150
• Default configuration: 2147483647

HBASE-ENV.SH configuration Specifies the system operating environment

exportJAVA_HOME=/usr/lib/jvm/java-6-sun/ #JDK HOMEexport HBASE_HOME=/home/hadoop/cdh4/hbase-0.94.2-cdh4.2.1 # HBase 安装目录export HBASE_LOG_DIR=/mnt/dfs/11/hbase/hbase-logs#日志输出路径

Tuning JVM Parameters

exportHBASE_OPTS="-verbose:gc -XX:+PrintGCDetails -Xloggc:${HBASE_LOG_DIR}/hbase-gc.log -XX:+PrintGCTimeStamps -XX:+PrintGCApplicationConcurrentTime -XX:+PrintGCApplicationStoppedTime \-server -Xmx20480m -Xms20480m -Xmn10240m -Xss256k  -XX:SurvivorRatio=4 -XX:MaxPermSize=256m -XX:MaxTenuringThreshold=15 \-XX:ParallelGCThreads=16 -XX:+UseConcMarkSweepGC -XX:+UseParNewGC  -XX:CMSFullGCsBeforeCompaction=5 -XX:+UseCMSCompactAtFullCollection \-XX:+CMSClassUnloadingEnabled  -XX:CMSInitiatingOccupancyFraction=70 -XX:+UseCMSInitiatingOccupancyOnly -XX:CMSMaxAbortablePrecleanTime=5000     \"

 

HBase parameter Configuration and description