HBase Performance Optimization

Query cache

Scanof theCachingthe default value of the property is1, meaning that each time the scanner RegionServer Crawl1record to match. We can setCachingas a ratio1a much larger value. For example, set to -, you can crawl -Note that the larger the value is, the more memory overhead the server will have.

Htableinterface htable=gethtable (tableName);

Scan scan= New Scan ();

/* Set Cache * /

Scan. setcaching (Staticconfig. Geticontrol_hbase_cache ());

Resultscanner scanner= Htable.getscanner (scan);

multithreaded configuration

Hbase.regionser.handler.count

The number of RPC listener instances in the Regionserver. For master, this attribute is the number of processing threads (handler) accepted by master. The default value is 10.

According to the business scenario of the regulatory layer, the matching query of 1 freight rates will produce 4 hbase concurrent queries. If there are 20, there may be 80 concurrent, this concurrency is equivalent. In addition to the appropriate adjustment of this parameter can increase the concurrent processing capacity, but also with the number of clusters and the configuration of the server directly, it is expected that the more clusters, the higher the number of server CPU, the more concurrent processing power.

Pre-partitioning

Hregion is the smallest unit of distributed storage and load balancing in HBase. The smallest unit means that different hregion can be distributed on different hregion servers. However, a hregion is not split across multiple servers.

Hbase.hregion.max.filesize

the maximum value of the hstorefile. If the storage file for any of the column families in region exceeds this limit, it will be split into two region. Default:268435456(256x1024x1024), which is 256M.

Our control file is relatively small, to reach the maximum partition limit of 256M need more control files. In order to increase the concurrency, we need to generate multiple hregion to save and process the data without reaching the partition cap, where the pre-partitioning functionality of hbase is used.

Example:

Configuration conf = hbaseconfiguration.create ()

Hbaseadmin admin = new hbaseadmin (conf);

htabledescriptor desc = New Htabledescriptor (

            bytes. tobytes (Table name

hcolumndescriptor coldef =   new   hcolumndescriptor (

Bytes. tobytes (colfamily));

Admin.createtable (DESC, bytes.tobytes (1L), Bytes.tobytes (10L), 10);

divide the area with the first character

Desc.setvalue (Htabledescriptor.split_policy,

KeyPrefixRegionSplitPolicy.class.getName ());

Desc.setvalue ("Prefix_split_key_policy.prefix_length", "1");

For more information, please follow: http://bbs.superwu.cn sweep Superman Academy QR code:

HBase Performance Optimization