HBase writes data, saves data, and reads data in a detailed process

Transferred from: http://www.aboutyun.com/thread-10886-1-1.html

After HBase 0.94 split strategy: http://www.aboutyun.com/thread-11211-1-1.html

What processes are required for 1.Client writing?
How does 2.Hbase read the data?






Client write-in Memstore, until Memstore full, flush into a storefile, up to a certain threshold, start compact merge operations Multiple StoreFile merged into one storefile, simultaneous version merging and data deletion, when the Storefiles compact is gradually formed, the larger StoreFile-a single storefile size exceeds a certain threshold value, Trigger split operation, the current region split into 2 region,region will be offline, the new split out of the 2 children region will be hmaster assigned to the corresponding hregionserver.


HBase writes data and saves data in the process

<ignore_js_op>

Client write-in Memstore, until Memstore full, flush into a storefile, up to a certain threshold, start compact merge operations Multiple StoreFile merged into one storefile, simultaneous version merging and data deletion, when the Storefiles compact is gradually formed, the larger StoreFile-a single storefile size exceeds a certain threshold value, Trigger split operation, the current region split into 2 region,region will be offline, the new split out of the 2 children region will be hmaster assigned to the corresponding hregionserver, So that the original 1 region of the pressure can be diverted to 2 region on this process, hbase is only to increase the data, the resulting update and delete operations are done in the compact phase, so that the user write operations only need to enter the memory to return immediately, thus ensuring I/O high performance.

Add to the above process:

Supplemental 1:hstore Storage is the core of hbase storage, which consists of two parts, part Memstore, and part storefiles. Supplementary 2:hlog function: in the Distributed system environment, cannot avoid the system error or the outage, once hregionserver exits, the memory data in the Memstore is lost, the introduction Hlog is to prevent this situation. Working mechanism: Each hregionserver will have a Hlog object, Hlog is a implementation of the write Ahead log class, each time the user action write Memstore, but also write a copy of the data to the Hlog file, hlog files regularly rolling out new, and delete the old files (data that has persisted to storefile). When the hregionserver unexpected termination, hmaster through zookeeper sense, hmaster first processing the legacy hlog files, the different region of the log data split, respectively, placed in the corresponding region directory, Then reassign the failed region (with the log that has just split) and pick up the hregionserver of these region in the process of load region, you will find that there is a history hlog need to deal with, so it will replay Hlog the data into Memstore, then flush to Storefiles to complete the data recovery. Supplement 3:region is storefiles,storefiles in the hfile composition, hfile in the data block of HBase, a data block inside there are many keyvalue pairs, each keyvalue in the value we need to save. Supplemental 4:<ignore_js_op>let's look at the picture above:A table, there are two column family (red color of one, yellow color of one), a column family has two columns, it can be seen that this is the largest feature of the column database, the same column family of data together, we also found that if there are more than one version, but also to save multiple versions. Finally, we also found that there is such a value: R1: Key value, cf1: Column family name, C1: listed. T1: Version number, value (the last picture illustrates where the value can be stored). Through this view, we found that if we design the table when we put these things: R1: Key value, cf1: Column family name, C1: The name of the list to take a short point is not we will save a lot of storage space!Also, we should get this understanding from this picture.: We look at the second-to-last graph, the efficiency of the field filtering is significantly reduced from left to right, so in the design of keyvalue users can consider to move some important filtering information to the left to the appropriate location, in order to improve the query performance without changing the amount of data. So simply put, the user should try to store the query dimension or information in the health, because it is the most efficient to filter the data.after getting the above understanding, we should have this kind of consciousness.: HBase data is stored sequentially in a specific scope because we store it in order, so it will always be stored in the same region, because one region can only be managed by one server, so we always add to the same area, Can cause a read-write hotspot to degrade the performance of the cluster. So the solution to this is still some, I can think of is, for example, we have 9 servers, then we go back to the current time, and then touch 9, add to the line health prefix, so it will be divided into different region on the average server, so that the benefit is because the connected data are distributed to different servers , the user can read the data in parallel in multiple threads, so that the throughput of the query increases. With regard to our version of control, we can either synchronize the time on multiple servers or simply set a client timestamp instead when the put is plugged into the data. (Because if we don't show the additions, we'll have to add our own time to our own server.) Supplement 5: When designing the table, there are two design methods, one is the high table design, the other is the Fat table design. According to HBase's split rules, our high-table design is easier to split (using key combinations), but if we design a fat table, and our fat data needs to be constantly modified, this design is reasonable, because our hbase guarantees the atomicity of the row-level, and if it is designed into a high table, it is not appropriate, Because there is no guarantee of atomicity across rows. Supplement 6: Write cache each put operation is actually RPC operation, it sends the client's data to the server and then returns, this is only suitable for small data volume operation, if an application needs to store thousands of rows of data per second into the HBase table, it is not appropriate to handle. The HBase API is equipped with a write buffer for the client, which collects the put operation and then calls the RPC operation to send the put to the server at once. By default, client buffers are disabled. You can activate the buffer by setting the auto-write to False. Table.setautoflush (false); void Flushcommits () throws IOException This method is forcing data to be written to the server. The user can also configure the client write buffer size according to the following method. void Setwritaebuffersize (Long writebuffersize) throws IOException, the default size is 2MB, this is also moderate, the average user inserts the data is not big, but if you insert the data large, You may want to consider increasing this value. This allows the client to perform a set of data in a single RPC request more efficiently. To each user's HTable setting a write buffer is also a hassle, in order to avoid trouble, users can set a large preset value in Hbase-site.xml.

1. <property>
3. <name>hbase.client.write.buffer</name>
5. <value>20971520</value>
7. </property>

Copy Code

Supplemental 7:hbase supports a large number of algorithms, and supports compression algorithms above the column family level, unless there are special reasons, otherwise we should try to use compression, compression usually results in better performance. With some tests, we recommend using the snappy algorithm to compress our hbase.

HBase Read data:

Client->zookeeper->. Root->. The meta-> user data sheet zookeeper recorded. Root path information (Root has only one region),. Root records the region information of the. Meta, (. Meta may have multiple region),. The meta contains information about region. Supplement 1: In HBase, all the storage files are divided into small blocks of storage that are loaded into memory when get or scan operations are similar to the storage unit pages in the RDBMS. The default size of this parameter is 64K. Set by: void Setblocksize (int s), (the default size of hfile in HBase is 64K with HDFs blocks is 64M OK) hbase sequentially reads a block of data into the memory cache, The number of disk I/O can be reduced effectively by reading adjacent data and then reading it in memory without having to read it again from disk. This parameter defaults to true, which means that every chunk read is cached in memory. However, if a particular column family is read sequentially by the user, it is better to set this property to FALSE, thereby preventing the use of caching fast. The reason for this description: If we access a particular column family, but we still enable this feature, this time our mechanism will load the data of our other unwanted column families into memory, increasing our burden, we use the condition is that we get adjacent data. void Setblockcacheenabled (Boolean blockcacheenable); Supplement 2:1: Disables Auto-write. We have a large number of data to be inserted, if we do not prohibit, put instance will be transferred to the Regio server one by one, if the user prohibits the automatic brush write function, put operation will be sent when the write buffer is filled. 2: Use scan cache. If HBase is used as the input source for a mapreduce job, it is best to set the cache with the Setcaching () method as the input scanner instance of the MapReduce job to a number larger than the default value of 1. Using the default value means that the map task will request a region server when processing each record. However, if this value is 500, then you can transfer 500 data to the client for processing, of course, the data is based on your situation. This is a row level, which is explained on our 119 page. 3: Limit the scan range. This is very well understood, such as we have to deal with a lot of rows (especially as the input source of MapReduce), in which we have scan.addfamily () when using scan, this time we just need toTo get to several columns in this column family, we must be precise. Because too many columns can lead to loss of efficiency. 4: Close Resultscanner Of course this does not improve our efficiency, but if it does not, it will have an effect on efficiency. 5: Block cache usage first of all our block caches are started by scan.setcachebolcks (), those frequently accessed rows we should use the cache block, but the MapReduce job uses scanning a large number of rows, we should not use this. (This block cache is not the same as the one I mentioned in the fourth section). 6: Optimize the way to get the line health of course, the premise of this is that we only need the table in the row health, to use. So how to use on page 411 has instructions. 7: Close the Wal book on the put, but I personally feel that this function is not good, because we shut down this feature, the server will not put put write to Wal, but directly to the memstore, so that once the server fails our data is lost.

HBase writes data, saves data, and reads data in a detailed process