HBase Memstore Flush

我之前一直以為HBase 只會對那些達到了閾值的Memstore進行flush，雖然知道有個配置參數hbase.hregion.memstore.flush.size是控制這個閾值的，但是沒想到level是HRegion，但是所有這些理解在讀了Visualizing HBase Flushes And Compactions給出的測試結果後才發現當初自己理解錯了，於是google一下，發現apache jire上有這樣一個ticket，提出的問題如下：

 

Today, the flush decision is made using the aggregate size of all column families. When large and small column families co-exist, this causes many small flushes of the smaller CF. We need to make per-CF flush decisions.

 

下面列出兩個比較有意思的comments供大家考：

 

Nicolas Spiegelberg

 

This is a substantial refactoring effort. My current development strategy is to break this down into 4 parts. Each one will have a diff + review board so you guys don't get overwhelmed...

1. move flushcache() from Region -> Store. have Region.flushcache loop through Store API
2. move locks from Region -> Store. figure out flush/compact/split locking strategy
3. refactor HLog to store per-CF seqnum info
4. refactor MemStoreFlusher from regionsInQueue to storesInQueue

 

Schubert Zhang

 

This jira is very useful in practice.
In HBase, the horizontal partitions by rowkey-ranges make regions, and the vertical partitions by column-family make stores. These horizontal and vertical partitoning schema make a data tetragonum — the store in hbase.

The memstore is base on the store, so the flush and compaction need also be based on store. The memstoreSize in HRegion should be in HStore.

For flexible configuration, I think we shlould be able to configure memstoresize (i.e. hbase.hregion.memstore.flush.size) in Column-Family level (when create table). And if possaible, I want the maxStoreSize also be configurable for different Column-Family.

 

可以看出hbase owner還是在Improvement這個問題，希望他們早日解決這個問題。當然我會再測試一下最新版本0.90的情況並結合代碼給一些分析出來。

 

https://issues.apache.org/jira/browse/HBASE-3149

 

---------------------------------------分割線----------------------------------------

經過測試hbase 0.90發現，Memstore flush的確是按照上述分析去做的，下面上傳的兩張圖都可以得出這個結論：

 

１）首先建立兩個CF： edp1, edp2

2 ）向edp1插入一條記錄，因為hbase.hregion.memstore.flush.size設定為64M，所以肯定不會發生flush，而且通過admin web去查看，的確如此。

3 ）向edp2插入大量記錄，足以會讓MemStore的大小超過設定的閾值，通過admin web去查看，有兩個storefiles存在了，說明edp2的memstore超過閾值會導致其他的memstore也會被flush到disk。

註：為什麼edp2會有兩個檔案，因為關閉hbase一會，它會將memstore中的資料flush到新的storefile中。注意檔案建立的時間，說明flush是在不同時間發生的。（疑問：為什麼edp2的storefile大小>64M呢。原因很簡單因為storefile除了要儲存memstore裡面的KeyValue，還要儲存meta block, data block index, meta block index, file info and fixed file trailer，額外的這些資訊佔用了其他部分的空間）

 

-------------------------------------第二次更新該知識點----------------------

1）Within one region, if the sum of the memstores of the column families reaches hbase.hregion.memstore.flush.size, then the memstores of that region will be flushed.

But within one region server, there can be many (hundreds, thousands) of regions active, and thus hbase.hregion.memstore.flush.size applies to each of these regions individually, not to the sum of them. That is where hbase.regionserver.global.memstore.upperLimit comes into play.

Normal 0 7.8 磅 0 2 false false false MicrosoftInternetExplorer4 /* Style Definitions */ table.MsoNormalTable{mso-style-name:普通表格;mso-tstyle-rowband-size:0;mso-tstyle-colband-size:0;mso-style-noshow:yes;mso-style-parent:"";mso-padding-alt:0cm 5.4pt 0cm 5.4pt;mso-para-margin:0cm;mso-para-margin-bottom:.0001pt;mso-pagination:widow-orphan;font-size:10.0pt;font-family:"Times New Roman";mso-ansi-language:#0400;mso-fareast-language:#0400;mso-bidi-language:#0400;}

2）Hbase memstore flush目前是使用單線程去幹活，這是一個系統瓶頸，所以不可避免的將會實現多線程flush來解決這個問題，下面是jira上面的一個ticket，有興趣仔細閱讀一下：

Priorities and multi-threading for MemStore flushing

Flushes are currently triggered when:

1.                     MemStore size of an HRegion is > hbase.hregion.memstore.flush.size (checked after every memstore insertion)

2.                     Sum of MemStores of an HRegionServer are > total.memstore.heap (checked after every memstore insertion)

3.                     Number of HLogs of an HRegionServer are > max.hlogs (checked after every hlog roll?) TODO: Verify

4.                     HRegion is being closed (when receiving message from master)

5.                     HRegionServer is being quiesced (when receiving message from master)

6.                     Client manually triggers flush of an HRegion (when receiving message from master)

7.                     MemStore size of an HRegion is > memstore.flush.size * hbase.hregion.memstore.multiplier (checked before every memstore insertion)

There are 3 different types of flushes that correspond to these 6 events:

1.                     Low Priority Flush This occurs in response to #1. This is the lowest priority flush and does not need any tricks. All other flush types should be completed before any of this type are done. The one optimization it has is that if it determines that a compaction would be triggered after the flush finished, it should cancel the flush and instead trigger a CompactAndFlush.

2.                     High Priority Flush This occurs in response to #2, #3, #6, and #7. High priority flushes occur in response to memory pressure, WAL pressure, or because a user has asked for the flush. These flushes should occur before any low priority flushes are processed. They are only special because of their priority, otherwise the implementation of the flush is identical to a low priority flush. This flush type explicitly does not contain the CompactAndFlush check because it wants to flush as fast as possible.

3.                     High Priority Double Flush TODO: Region closing currently does flushing in-band rather than through the flush queue. Should move those into using handlers once we have blocking call. Therefore, double-flush priority is not currently used. TODO: Do we want a separate priority here once we do use this for closes? Or are they just high priority flushes? The first one is even a low priority flush, second one high priority? This occurs in response to #4. When an HRegion is being closed (but not when a cluster is being quiesced) we want to minimize the amount of time the region is unavailable. To do this we do a double flush. A flush is done, then the region is closed, then an additional flush is done before the region is available to be re-opened. This should happen when the Master asks a region to close because of reassignment. This may also occur before splits to reduce the amount of time the parent is offline before the daughters come back online.