Tuning of HBase

1. The design of the table

1.1 pre-creating regions By default, a region partition is created automatically when the HBase table is created, and when the data is imported, all HBase clients write the data to this region until the region is large enough to be split.

One way to speed up batch writes is to create some empty regions, so that when data is written to HBase, the data is load-balanced within the cluster according to region partitioning.

Here is an example:

public static Boolean createtable (hbaseadmin admin, htabledescriptor table, byte[][] splits) throws IOException {

try {admin.createtable (table, splits); return true;} catch (Tableexistsexception e) {logger.info ("table" + Table.getna Measstring () + "already exists"); The table already exists return false; } public static byte[][] Gethexsplits (String startkey, string endkey, int numregions) {//start:001,endkey:100,10region [001,010] [011,020] byte[][] splits = new byte[numregions-1][];

BigInteger Lowestkey = new BigInteger (Startkey, 16);

BigInteger Highestkey = new BigInteger (EndKey, 16); BigInteger range = highestkey.subtract (Lowestkey); BigInteger regionincrement = Range.divide (biginteger.valueof (numregions)); Lowestkey = Lowestkey.add (regionincrement); for (int i=0 i < numregions-1;i++) {BigInteger key = Lowestkey.add regionincrement.multiply (biginteger.valueof (i)) ; Byte[] B = String.Format ("%016x", Key). GetBytes (); Splits[i] = b; return splits; }

1.2 Row Key

HBase The row key is used to retrieve records in a table, supporting the following three ways: 

Access through a single row key: That is, a get operation is performed according to a row key value; 

Scan through the range of row key: That is, by setting Startrowkey and Endrowkey, scanning within this range; 

Full table Scan: Directly scans all row records in the entire table.

In HBase, the row key can be any string, the maximum length 64KB, the actual application is generally 10~100bytes, save as a byte[] byte array, generally designed to be fixed-length. The row key is stored in a dictionary order, so when you design row key, take advantage of this sorting feature to store the data that is often read together in a piece of data that might be accessed recently. For example: If the most recent data written in the HBase table is most likely to be accessed, consider using the timestamp as part of the row key, and because it is a dictionary order, you can use Long.max_value-timestamp as the row key. This ensures that the newly written data can be hit quickly when read.

Rowkey:1, size the smaller the better 2, the value according to the functional requirements of 3, row best have hashing principle.

A) counter

b) Hash 001-020 001 100 002 200 010 010 011 110

1.3 Column Family

Do not define too many column family in a table. Currently HBase does not work well with tables that exceed 2~3 column family. Because a column family is flush, its adjacent column family is triggered flush by the association effect, which eventually results in more I/O being generated by the system.

1.4 in Memory

When creating a table, you can place the table in the Regionserver cache by Hcolumndescriptor.setinmemory (true) to ensure that it is cached when read.

1.5 Max Version

When you create a table, you can set the maximum version of the data in the table through hcolumndescriptor.setmaxversions (int maxversions), and if you only need to save the latest version of the data, you can set Setmaxversions (1).

1.6 Time to Live

When you create a table, you can set the storage lifetime of the data in the table through hcolumndescriptor.settimetolive (int timetolive), and the expired data is automatically deleted, for example, if you only need to store data for the last two days, Then you can set Settimetolive (2 * 24 * 60 * 60).

1.7 Compact & Split

In HBase, the data is first written to the Wal log (Hlog) and memory (Memstore) in the update, and the data in the Memstore is sorted.when the memstore accumulates to a certain threshold, a new memstore is created and the old Memstore is added to the flush queue, flush from a separate thread to disk and becomes a storefile. At the same time, the system will record a redo point in the zookeeper, indicating that the previous change has persisted (minor compact).StoreFile is read-only and can no longer be modified once it is created. So the hbase update is actually a continuous operation. When the number of storefile in a store reaches a certain threshold, it will be merged (Major compact), the modification of the same key will be merged together to form a large storefile, when the storefile size reaches a certain threshold, it will StoreFile (split) is divided into two storefile. Because updates to the table are constantly appended, when processing a read request, you need to access all the StoreFile and Memstore in the store and merge them according to the row key, because StoreFile and Memstore are sorted,and StoreFile with an in-memory index, the process is usually relatively fast。 In practice, it is possible to consider manually major compact when necessary, merging the modification of the same row key to form a large storefile. At the same time, the storefile can be set larger, reduce the occurrence of split. HBase in order to prevent small files (to be brushed to disk menstore) too much to ensure that query efficiency, hbase need to be in the necessary time to merge these small store file into a relatively large store file, this process is called compaction.in HBase, there are two main types of Compaction:minor compaction and major compaction. Minor compaction: A combination of smaller, fewer files. Major compaction's function is to combine all store file into one, triggering major compaction possible conditions: major_compact command, Majorcompact () API, Region Server Autorun (Related parameters: Hbase.hregion.majoucompaction defaults to 24 hours, hbase.hregion.majorcompaction.jetter defaults to 0.2 to prevent region server Major compaction at the same time). The function of the Hbase.hregion.majorcompaction.jetter parameter is to float the value of the parameter hbase.hregion.majoucompaction, if two parameters are the default values 24 and 0, 2, then the major compact eventually uses the value: 19.2~28.8 this range. 1, turn off automatic major compaction 2, manual programming major compaction Timer class, Contab minor operation mechanism to be more complicated, it is decided by a few parameters together: Hbase.hstore.compaction.min: The default value is 3, which means that minor compaction will not start until at least three store file satisfies the condition Hbase.hstore.compaction.max the default is 10 That represents a maximum of 10 store file selections in a minor compaction

Hbase.hstore.compaction.min.size indicates that store file with a file size greater than this value must be added to the store file in minor compaction Hbase.hstore.compaction.max.size indicates that the file size is greater than the value of the store file must be minor compaction excluded Hbase.hstore.compaction.ratio will store Files are sorted by file age (older to younger), minor compaction always choose from older store file The following is the second part of this article: Write table operations related optimization methods.

2. Write table operation

More than 2.1 htable concurrent write to create multiple htable clients for write operations, improve the throughput of write data, an example: Static final Configuration conf = Hbaseconfiguration.create (); Static final String table_log_name = "User_log"; Wtablelog = new Htable[tablen]; for (int i = 0; i < Tablen i++) {Wtablelog[i] = new htable (conf, table_log_name); Wtablelog[i].setwritebuffersize (5 * 1024 * 1024); 5MB Wtablelog[i].setautoflush (FALSE); }
2.2 Htable parameter setting
2.2.1 Auto Flush The automatic Flush of the htable write client can be turned off by calling the Htable.setautoflush (false) method so that the data can be written to the hbase in batches, rather than having a put to perform an update. Write requests are actually initiated to the HBase server only when put fills the client write cache. Auto Flush is turned on by default.
2.2.2 Write Buffer
By calling the Htable.setwritebuffersize (writebuffersize) method, you can set the write buffer size of the htable client, and if the new buffer is less than the data in the current write buffer, The buffer will be flush to the service side. Where the units of the Writebuffersize are byte bytes, the value can be set based on how much data is actually written.
2.2.3 WAL Flag in Hbae, when the client submits data to Regionserver in the cluster (Put/delete operation), first writes the WAL (write ahead log) log (that is, Hlog, All region on a regionserver share a hlog), only after the Wal log succeeds, then the Memstore is written, and the client is notified that the submission is successful, and if the write Wal log fails, the client is notified that the submission failed. The advantage of this is that you can do the data recovery after regionserver downtime. Therefore, for relatively less important data, you can discard the write Wal log by calling the Put.setwritetowal (false) or Delete.setwritetowal (false) function at the put/delete operation. Thus improving the performance of data writes. It is worth noting that the Wal log is carefully chosen to close because, once regionserver is down, the Put/delete data will not be recoverable according to the Wal log. 2.3 Batch write by invoking the Htable.put (Put) method, you can write a specified row key record to HBase, and HBase provides another method: by calling the Htable.put (list<put>) method, you can add the specified row Key list, bulk write multiple line records, the benefit of this is bulk execution, only need one network I/O overhead, which for the real-time data requirements, network transport RTT High scenario can bring significant performance improvement.
More than 2.4 threads concurrent write on the client to open multiple htable write threads, each write thread is responsible for a Htable object flush operation, this combination of timed flush and write buffer (writebuffersize), can ensure that when the amount of data is small, Data can be flush in a relatively short period of time (such as 1 seconds), while ensuring that when the volume of data, write a buffer full time to flush. Here's a concrete example: for (int i = 0; i < Threadn i++) {thread th = new Thread () {public void run () {while (true) {try {Slee P (1000); 1 second} catch (Interruptedexception e) {e.printstacktrace ();} synchronized (Wtablelog[i]) {
try {wtablelog[i].flushcommits ();} catch (IOException e) {e.printstacktrace ()}} } } }; Th.setdaemon (TRUE); Th.start (); }
Summary of HBase Performance optimization Method (III.): Read table operation This paper summarizes several commonly used performance optimization methods from the angle of hbase application design and development. About the HBase system configuration level optimization, can refer to: Taobao Ken Wu classmate's blog. The following is a summary of the third part of this article: Reading table operation related optimization methods.
3. Read Table operation
More than 3.1 htable concurrent read to create multiple htable clients for read operations, improve the throughput of read data, one example: Static final Configuration conf = Hbaseconfiguration.create (); Static final String table_log_name = "User_log"; Rtablelog = new Htable[tablen]; for (int i = 0; i < Tablen i++) {Rtablelog[i] = new htable (conf, table_log_name); rtablelog[i].setscannercaching (50); }
3.2 htable parameter setting
3.2.1 Scanner Caching
The Hbase.client.scanner.caching configuration item can set the number of data bars that hbase scanner to crawl from the server at a time, by default one time. By setting it to a reasonable value, you can reduce the time cost of next () in the scan process, at the cost of scanner to maintain these cache-line records through the client's memory. There are three places to configure: 1 To configure in the HBase conf profile, 2 to configure by calling htable.setscannercaching (int scannercaching), 3 by calling Scan.setcaching (int caching) for configuration. The priority of the three is getting higher.
3.2.2 Scan Attribute Selection Scan Specifies the required column Family to reduce the amount of data transferred by the network, otherwise the default Scan operation returns data for all column Family in the entire row.
3.2.3 Close Resultscanner After you have finished fetching the data by scan, remember to turn off Resultscanner, otherwise the regionserver may be problematic (the corresponding server resource cannot be freed).
3.3 Batch read by calling the Htable.get (get) method to obtain a row of records based on a specified row key, the same HBase provides another method: by invoking the Htable.get (list<get>) method, you can base a specified row The key list, the bulk of the multiple-line records, the benefit of this is bulk execution, only one network I/O overhead, which can bring significant performance gains in situations where data is required for real time and network traffic RTT is high.
More than 3.4 threads read concurrently to open multiple htable read threads on the client, and each read thread is responsible for get operations through the Htable object. The following is a multithreaded concurrent read HBase, get the store one day for each minute PV value example: public class Datareaderserver {//Get the store every minute of the day PV Value entry function public static Concurrenthashmap<string, string> Getunitminutepv (Long uid, long startstamp, long Endstamp) {long min = Startstamp; int count = (int) ((endstamp-startstamp)/(60*1000)); list<string> lst = new arraylist<string> (); for (int i = 0; I <= count; i++) {min = startstamp + i * 1000; Lst.add (uid + "_" + min);} return Parallelbatchmi NUTEPV (LST);
}//Multithreading concurrent query, get the minute PV value private static concurrenthashmap<string, string> Parallelbatchminutepv (list<string> Lstkeys) {concurrenthashmap<string, string> Hashret = new concurrenthashmap<string, String> (); int parallel = 3; List<list<string>> Lstbatchkeys = null; if (Lstkeys.size () < parallel) {Lstbatchkeys = new arraylist<list<string>> (1); Lstbatchkeys.add (LstKeys ); } else{Lstbatchkeys = new arraylist<list<string>> (parallel); for (int i = 0; i < parallel; i++) {List<s tring> lst = new arraylist<string> (); Lstbatchkeys.add (LST); for (int i = 0; i < lstkeys.size (); i + +) {Lstbatchkeys.get (I%parallel). Add (Lstkeys.get (i));} list<future< concurrenthashmap<string, string> >> futures = new arraylist<future< Concurrenthashmap<string, String> >> (5); Threadfactorybuilder builder = new Threadfactorybuilder (); Builder.setnameformat ("Parallelbatchquery"); Threadfactory factory = bUilder.build (); Threadpoolexecutor executor = (threadpoolexecutor) executors.newfixedthreadpool (Lstbatchkeys.size (), factory); for (list<string> Keys:lstbatchkeys) {callable< concurrenthashmap<string, string> > Callable = new Bat Chminutepvcallable (keys); futuretask< concurrenthashmap<string, string> > Future = (futuretask< concurrenthashmap<string, string> >) executor.submit (callable); Futures.add (future); }
Executor.shutdown (); Wait for all the tasks to finish try {Boolean stillrunning =!executor.awaittermination (5000000, Timeunit.millisecond S); if (stillrunning) {try {executor.shutdownnow ();} catch (Exception e) {//TODO auto-generated catch block E.printstackt Race (); (interruptedexception e) {try {thread.currentthread (). Interrupt ()} catch (Exception E1) {//TODO Auto-gen Erated Catch block E1.printstacktrace (); Exception for (Future f:futures) {try {if (f.get ()!= null) {Hashret.putall (concurrenthashmap< String, string>) f.get ()); } catch (Interruptedexception e) {try {thread.currentthread (). interrupt ();} catch (Exception E1) {//TODO Auto-gener Ated Catch block E1.printstacktrace (); The catch (Executionexception e) {e.printstacktrace ()}
return hashret; ///One thread bulk query, get the minute PV value protected static concurrenthashmap<string, string> Getbatchminutepv (list<string> Lstkeys) {concurrenthashmap<string, string> hashret = null; list<get> lstget = new arraylist<get> (); string[] SplitValue = null; for (String s:lstkeys) {splitvalue = S.split ("_"); Long uid = Long.parselong (splitvalue[0)); Long min = Long.parselong ( SPLITVALUE[1]); byte[] key = new BYTE[16]; Bytes.putlong (key, 0, UID); Bytes.putlong (Key, 8, Min); Get g = new Get (key); g.addfamily (FP); Lstget.add (g); } result[] res = NULL; try {res = Tableminutepv[rand.nextint (tablen)].get (lstget);} catch (IOException E1) {logger.error ("Tableminutepv excep tion, e= "+ e1.getstacktrace ()); } if (res!= null && res.length > 0) {hashret = new concurrenthashmap<string, string> (res.length); for ( Result Re:res {if (re!= null &&!re.isempty ()) {try {byte[] key = Re.getrow (); byte[] value = Re.getvalue (f P, CP); if (Key!= null&& value!= null) {Hashret.put (string.valueof (Bytes.tolong (Key, Bytes.sizeof_long))
String.valueof (Bytes. Tolong (value))); } catch (Exception E2) {Logger.error (E2.getstacktrace ());} } return Hashret; }//Call interface class to implement callable interface class Batchminutepvcallable implements Callable<concurrenthashmap<string, string> >{private list<string> keys; public batchminutepvcallable (list<string> lstkeys) {This.keys = LstKeys;} Public concurrenthashmap<string, String> call () throws Exception {return Datareadserver.getbatchminutepv (keys); }
3.5 Cached query results for frequent query HBase application scenarios, you can consider caching in the application, when there is a new query request, first in the cache to find, if there is a direct return, no longer query hbase, otherwise the HBase initiated read request query, The query results are then cached in the application. As for the cache replacement strategy, you can consider common strategies such as LRU.
3.6 Blockcache HBase on the regionserver memory is divided into two parts, part as a memstore, mainly used for writing, the other part as a blockcache, mainly for reading. Write requests will be written first memstore,regionserver will give each region a memstore, and when Memstore full 64MB, will start flush refresh to disk. When the total size of the memstore exceeds the limit
(HeapSize * hbase.regionserver.global.memstore.upperLimit * 0.9) will force the flush process to start, flush from the largest memstore until it is below the limit. Read the request first to Memstore to check the data, can not find the Blockcache to check, and then find the disk will be read, and the results of the reading into the Blockcache. Since Blockcache adopts the LRU strategy, Blockcache reaches the upper limit (heapsize * hfile.block.cache.size * 0.85), it will start the elimination mechanism, eliminating the oldest batch of data. A regionserver has a blockcache and N Memstore, and their size cannot be greater than or equal to heapsize * 0.8, otherwise the hbase cannot start. The default Blockcache is 0.2, while Memstore is 0.4. For systems that pay attention to read response time, you can set the Blockcache larger, such as setting blockcache=0.4,memstore=0.39 to increase the cache hit rate. For the Blockcache mechanism, refer to the blockcache mechanism of the block cache,hbase of HBase, the calculation and use of caching in HBase.
Htable and Htablepool usage considerations htable and Htablepool are part of the HBase client API that you can use to crud the HBase tables. The following combined with the application in the project, the use of the two processes of attention to do a summary summary.
Configuration conf = hbaseconfiguration.create ();
Try (Connection Connection = connectionfactory.createconnection (conf)) {
Try (Table table = connection.gettable (tablename.valueof (tablename)) {
Use table as needed, the table returned is lightweight
}
}
4.HTable Htable is a Java API object that HBase client communicates with the HBase server, and the client can perform CRUD operations on Htable object and server (add-check). It is very simple to create: Configuration conf = Hbaseconfiguration.create (); htable table = new htable (conf, "TableName"); TODO CRUD Operation ... Some considerations when using the Htable: 1. Circumvent the creation cost of Htable objects
Because the client creates a Htable object, a series of actions are required: check. META. The table confirms that the HBase table with the specified name exists, that the table is valid, and so on, that the overall time overhead is heavier and can take several seconds, so it is a good idea to create the Htable object that is needed at the start of the program, if you use the Java API, which is typically created Direct reuse after program startup. 2. The Htable object is not a thread-safe htable object that is not thread-safe for the client to read and write data, so when multithreading, create a duplicate htable object for each thread, and do not share htable objects among the different objects, especially in the client auto When Flash is set to false, the presence of local write buffer may result in inconsistent data. 3. Shared configuration Htable object sharing configuration objects between htable objects, which benefits from  Shared zookeeper connections: Each client needs to establish a connection with zookeeper to query the user's table Regions position, this information can be cached after the connection is established to share the use;  Shared public resources: clients need to find-root-and. META. Tables through zookeeper, which requires network transport overhead, and the client caches these public resources to reduce subsequent network transport overhead and speed up the lookup process. Thus, compared to the following: htable table1 = new Htable ("Table1"); htable table2 = new Htable ("Table2"); The following way is more effective: Configuration conf = Hbaseconfiguration.create (); htable table1 = new htable (conf, "Table1"); htable table2 = new htable (conf, "table2"); Note: Even high-load multithreaded programs do not find performance problems due to shared configuration, and if not, you can try not to share configuration.
5.HTablePool Htablepool can solve the problem of thread insecurity in htable, while maintaining a fixed number of htable objects to reuse these htable resource objects during program operation. Configuration conf = hbaseconfiguration.create (); Htablepool pool = new Htablepool (conf, 10); 1. Htablepool can automatically create Htable objects, and is completely transparent to the client, to avoid the problem of concurrency modification between multithreading. 2. The Htable object in Htablepool is a public configuration connection that can reduce network overhead.
The use of Htablepool is simple: each time before the operation, through the Htablepool GetTable method to obtain a Htable object, and then put/get/scan/delete and other operations, Finally, the Htable object is put back into Htablepool through the Htablepool puttable method. Here is a simple example of using Htablepool: public void CreateUser (string username, string firstName, String lastName, string email, string pas Sword, String roles) throws IOException {htable table = rm.gettable (usertable.name); Put on = new put (Bytes.tobytes (username)); Put.add (usertable.data_family, Usertable.firstname, Bytes.tobytes (FIRSTNAME)); Put.add (usertable.data_family, Usertable.lastname, Bytes.tobytes (LASTNAME)); Put.add (usertable.data_family, Usertable.email, Bytes.tobytes (EMAIL)); Put.add (usertable.data_family, usertable.credentials, bytes.tobytes (password)); Put.add (usertable.data_family, Usertable.roles, Bytes.tobytes (ROLES)); Table.put (Put); Table.flushcommits (); Rm.puttable (table); }
HBase and DBMS comparisons:
The query data is not flexible:
1, can not use between column filter query
2. Full-text indexing is not supported. Complete Full-text search using SOLR and hbase consolidation.
(a) Use Mr to read the HBase data in bulk, and create an index (no store) in SOLR to store the Rowkey value.
(b) Search for Rowkey (pagination) from the index according to the keyword
c) query all data from HBase according to Rowkey