HBase Memstore Flush

I always thought hbase would only flush the memstore that reached the threshold, Although it is known that there is a configuration parameter hbase.hregion.memstore.flush.size that controls this threshold, it is not thought that level is hregion, but all of these understandings are read visualizing HBase flushes and Compactions gave the test results only found that the original understanding of the wrong, so Google, and found that Apache Jire have such a ticket, asked the following questions:

Today, the flush decision are made using the aggregate size of all column families. When large and small column families co-exist, this is causes many small of the flushes CF. We smaller to make need FL Ush decisions.

Here are two more interesting comments for you to test:

Nicolas Spiegelberg

This is a substantial refactoring effort. The My current development strategy are to break the to 4 parts. Each one would have a diff + review board so guys don t get overwhelmed ...

1. Move flushcache () from Region-> Store. Have Region.flushcache Loop through Store API
2. Move locks from Region-> Store. Figure out Flush/compact/split locking strategy
3. Refactor Hlog to store PER-CF seqnum info
4. Refactor Memstoreflusher from Regionsinqueue to Storesinqueue

Schubert Zhang

This jira is very useful in practice.
In HBase, the horizontal partitions by rowkey-ranges make regions, and the vertical from partitions make store S. These horizontal and vertical partitoning schemas make a data tetragonum-the store in HBase.

The Memstore is base in the store, so the flush and compaction need also are based on store. The memstoresize in Hregion should is in Hstore.

For flexible configuration, I-I-we shlould be able to configure memstoresize (i.e. hbase.hregion.memstore.flush.size) In column-family level (when CREATE table). And if possaible, I want the maxstoresize also is configurable for different.

Can see HBase owner or in improvement this problem, hope they solve this problem at an early date. Of course I will test the latest version 0.90 and combine the code to give some analysis.

https://issues.apache.org/jira/browse/HBASE-3149

---------------------------------------Split Line----------------------------------------

After testing hbase 0.90 found, Memstore Flush is indeed based on the above analysis to do, the following uploaded two pictures can come to this conclusion:

1 first create two cf:edp1, EDP2

2 Insert a record into the EDP1, because the hbase.hregion.memstore.flush.size is set to 64M, so it certainly does not happen flush, and it is viewed through the admin web, indeed.

3 Insert a large number of records to EDP2, enough to allow the size of the Memstore to exceed the set threshold, through the admin web to view, there are two storefiles exist, This indicates that the memstore exceeding the threshold value of the EDP2 will cause other memstore to be flush to disk.

Note: Why EDP2 has two files, because closing hbase a while, it will flush the data in Memstore to the new storefile. Note the time the file was created, indicating that flush occurred at different times. (question: Why EDP2 's storefile size >64m?) The reason is simple because storefile to save the keyvalue in Memstore, you save the meta block, the data Block index, the Meta block index, the file info and the fixed file Trai Ler, the extra information takes up other parts of the space.

-------------------------------------Update the knowledge point for the second time----------------------

1) Within One region, if the sum of the memstores of the column families reaches Hbase.hregion.memstore.flush.size, then th e Memstores of that region would be flushed.

But within one region server, there can be many (hundreds, thousands) of regions active, and thus hbase.hregion.memstore.f Lush.size applies to each of this regions individually, not to the sum of them. Where Hbase.regionserver.global.memstore.upperLimit comes into play.

Normal 0 7.8 lb 0 2 false false MicrosoftInternetExplorer4/* Style definitions/table. msonormaltable {mso-style-name: plain form; mso-tstyle-rowband-size:0; mso-tstyle-colband-size:0; mso-style-noshow:yes; Mso-style-parent: ""; mso-padding-alt:0cm 5.4pt 0cm 5.4pt; mso-para-margin:0cm; mso-para-margin-bottom:.0001pt; Mso-pagination:widow-orphan; font-size:10.0pt; Font-family: "Times New Roman"; Mso-ansi-language: #0400; Mso-fareast-language: #0400; Mso-bidi-language: #0400;}

2 Hbase Memstore Flush is currently using a single thread to work, this is a system bottleneck, so the inevitable will be implemented multithreading flush to solve this problem, the following is Jira above a ticket, interested in reading carefully:

Priorities and multi-threading for Memstore Flushing

Flushes are currently triggered when:

1. Memstore size of an hregion are > hbase.hregion.memstore.flush.size (checked after every Memstore Insertion)

2. Sum of Memstores of Hregionserver are > total.memstore.heap (checked after every memstore ins Ertion)

3. Number of Hlogs of Hregionserver are > max.hlogs (checked after every hlog roll?) Todo:verify

4. Hregion is being closed (while receiving message from master)

5. Hregionserver is being quiesced (while receiving message from master)

6. Client manually triggers flush of a hregion (when receiving message from master)

7. Memstore size of a hregion is > memstore.flush.size * hbase.hregion.memstore.multiplier (Checke D before every memstore insertion)

There are 3 different types of flushes that correspond to these 6 events:

1. Priority Flush This occurs in response to #1. This is the lowest priority flush and does not need any tricks. All of the other flush types should is completed before any of the this type are done. The one optimization it has is this if it determines that a compaction would to triggered after the flush finished, it sho Uld cancel the flush and instead trigger a compactandflush.

2.                      High Priority Flush This occurs in response to #2, #3, #6, and #7. High priority flushes occur into response to memory pressure, WAL pressure, or because a user has for the asked. These flushes should occur before no low priority flushes are. They are only special because of their priority, otherwise the implementation of flush of Flush. This flush type explicitly does is not contain the Compactandflush check because it wants to flush as fast as possible.

3.                      High Priority Double Flush todo:region closing currently does flushing In-band rather than through the Flush Queue. Should move those to using handlers once we have call. Therefore, Double-flush priority is not currently used. Todo:do we want a separate priority here once we did use this for closes? Or are they just high priority flushes? The one is even a low priority flush, second one high priority? This is occurs in response to #4. When a hregion was being closed (but not when a cluster was being quiesced) we want to minimize the amount of the Regi On is unavailable. To does this we do a double flush. A Flush is done, then the region are closed, then an additional flush are done before the region are available to be re-opene D. This should happen is the Master asks a region to close because of reassignment. This could also occur before splits to reduCe the amount of time the ' parent is offline before ' daughters come back online.