HBase Compaction Process Analysis

The following analysis is based on the HBase-0.98 version.

Hregionserver starts, the Compactionchecker thread is started, andCompactionchecker detects if the region requires compaction.

The main implementation logic is as follows:

protected void chore () {for (hregion r:this.instance.onlineregions.values ()) {if (r = = NULL) cont        Inue;            For (Store s:r.getstores (). values ()) {try {long multiplier = S.getcompactioncheckmultiplier ();            Assert multiplier > 0;            If (iteration% multiplier! = 0) Continue; if (S.needscompaction ()) {//Queue a compaction.              Would recognize if major is needed.            This.instance.compactSplitThread.requestSystemCompaction (R, S, GetName () + "requests compaction"); } else if (S.ismajorcompaction ()) {if (majorcompactpriority = = Default_priority | | m  Ajorcompactpriority > R.getcompactpriority ()) {this.instance.compactSplitThread.requestCompaction (R, S, GetName () + "requests major compaction;              Use the default priority ", NULL); } else {This.instance.compactSplitThread.requestCompaction (R, S, GetName () + "requests major compaction;              Use configured Priority ", this.majorcompactpriority, NULL);          }}} catch (IOException e) {Log.warn ("Failed major compaction check on" + R, E); }}} iteration = (iteration = = Long.max_value)?    0: (iteration + 1); }  }

such as code, traverse Onlineregions, get each region's store to judge, where the needscompaction logic is as follows:

Public boolean needscompaction (final collection<storefile> storefiles,      final list<storefile> filescompacting) {    int numcandidates = storefiles.size ()-filescompacting.size ();    return numcandidates >= comconf.getminfilestocompact ();  }

The minfilestocompact is controlled by Hbase.hstore.compaction.min (the old version is: hbase.hstore.compactionThreshold), the default value is 3, That is, the number of storefiles below the store minus the number being compaction >=3 Yes, Compactionis required.

when Needscompaction is true, the Compactsplitthread.requestsystemcompaction method is called and the compaction request is sent, and the details are Compactsplitthread line Thread Analysis .

Needscompaction is false, it will determine whether it is ismajorcompaction, the specific logic is as follows:

/* * @param filestocompact Files to compact.   Can be null.   * @return True If we should run a major compaction. */Public Boolean ismajorcompaction (final collection<storefile> filestocompact) throws IOException {Boolea    n result = false;    Long mctime = Getnextmajorcompacttime (filestocompact);    if (filestocompact = = NULL | | filestocompact.isempty () | | mctime = = 0) {return result; }//Todo:use better method for determining stamp of last major (HBASE-2990) long Lowtimestamp = Storeutils.getlowe    Sttimestamp (filestocompact);    Long now = System.currenttimemillis ();      if (Lowtimestamp > 0l && lowtimestamp < (now-mctime)) {//Major compaction time has elapsed.      Long Cfttl = This.storeConfigInfo.getStoreFileTtl ();        if (filestocompact.size () = = 1) {//single file StoreFile SF = Filestocompact.iterator (). Next ();        Long Mintimestamp = Sf.getminimumtimestamp ();     Long oldest = (Mintimestamp = = null)       ?        Long.min_value:now-mintimestamp.longvalue (); if (Sf.ismajorcompaction () && (Cfttl = = Hconstants.forever | | Oldest < CFTTL)) {if (log.is Debugenabled ()) {Log.debug ("skipping major compaction of" + this + "because one (major) Compa          CTED file only and Oldesttime "+ oldest +" MS is < ttl= "+ Cfttl); }} else if (cfttl! = Hconstants.forever && Oldest > Cfttl) {log.debug ("Major compaction Trigg Ered on the store "+ this +", because keyvalues outdated;          Time since last major compaction "+ (Now-lowtimestamp) +" MS ");        result = true;              }} else {if (log.isdebugenabled ()) {Log.debug ("Major compaction triggered on store" + This + ";        Time since last major compaction "+ (Now-lowtimestamp) +" MS ");      } result = true;  }} return result; }

Not to be continued ...

HBase Compaction Process Analysis