Hregionserver Flush Operation Source Code Analysis

The flush operation is the process of saving hbase data to the hard disk, the specific flush flowchart is as follows, this article mainly briefly analyzes the process of the flush of the relevant source code.

Flush Task Commit

Whenever Hregion completes the operation of the data insert, it checks whether a flush,flush is required at this time to store the Hregion cached data to disk:

long addedSize = doMiniBatchMutation(batchOp);longthis.addAndGetGlobalMemstoreSize(addedSize);if (isFlushSize(newSize)) {    requestFlush();}

This paper mainly analyzes the process of flush and related data structure, which is called inside Requestflush:
this.rsServices.getFlushRequester().requestFlush(this);The actual MemStoreFlusher implementation of the flush operation is called:

 public  void  requestflush  (hregion r) { (regionsinqueue) {if  (!regionsinqueue.containskey (R)) { //this entry have no delay so it'll be added at the top of the flush  //queue.        It ' ll come out near immediately.         Flushregionentry fqe = new  flushregionentry (R);        this . Regionsinqueue.put (R, FQE);      this . Flushqueue.add (FQE); }    }  }

Memstoreflushrequeter has two data structure managers need to flush the task, private BlockingQueue<FlushQueueEntry>flushQueue Map<HRegion, FlushRegionEntry> regionsInQueue flushqueue equivalent to need to flush the work queue, The regionsinqueue is the information of the region that is already in the queue, and the code above indicates that when the region where the current request flush is not recorded in Flushqueue, it is added. Where Flushregionentry is a flushqueue unit data structure
Here the request for flush requests is completed, and then the Flushhander thread in Memstore waits for the region to take out and perform the flush task.

Flush tasks to perform pre-preparation

1.FlushHandler Remove the flushregionentry from the Flushqueue and perform
flushRegion(final FlushRegionEntry fqe)
This first determines if there are too many storefile files in the current region, and if so, you need to first storefile the merge operation (it is necessary to explain the data organization in Hregion), and then rejoin the queue. Otherwise, flush operations are performed directly on region:

isTooManyStoreFiles(region)this.server.compactSplitThread.requestSystemCompaction(                  region, Thread.currentThread().getName());                        this.flushQueue.add(fqe.requeue(this100));elsereturnfalse);

The main execution logic inside the 2.flushRegion function is as follows, first notifyflushrequest only to do some statistics on the number of flush threads, region.flashcache specifically responsible for the work of flush. After execution, related auxiliary actions are made based on the return value

Notifyflushrequest (region, Emergencyflush); Hregion.flushresult Flushresult = Region.flushcache ();BooleanShouldcompact = flushresult.iscompactionneeded ();//We just want to check the size BooleanShouldsplit = Region.checksplit ()! =NULL;if(Shouldsplit) { This. Server.compactSplitThread.requestSplit (region); }Else if(shouldcompact) {server.compactSplitThread.requestSystemCompaction (region, Thread.CurrentThread (). GetName ());}if(Flushresult.isflushsucceeded ()) {LongEndTime = Environmentedgemanager.currenttime (); Server.metricsRegionServer.updateFlushTime (endtime-starttime);}

Flush's task execution process

Flushcahe internal call Flushresult fs = Internalflushcache (status); actual flush operation, storeflushcontext implementation is Storeflusherimpl, Build a Storeflusherimpl for each hstore, which performs specific non-flush operations for the corresponding Hstore. The specific implementation of flush consists of three steps:
1. Snapshots

publicvoidprepare() {      this.snapshot = memstore.snapshot();      this.cacheFlushCount = snapshot.getCellsCount();      this.cacheFlushSize = snapshot.getSize();      new ArrayList<Path>(1);    }

2. Write the data in the Memestore to the. tmp file

   publicvoidflushCachethrows IOException {      tempFiles = HStore.this.flushCache(cacheFlushSeqNum, snapshot, status);    }

3. Write the. tmp file to the corresponding file in the corresponding CF and save the corresponding hfile file information with StoreFile.

     Public Boolean Commit(Monitoredtask status)throwsIOException {if( This. Tempfiles = =NULL|| This. Tempfiles.isempty ()) {return false; } list<storefile> storefiles =NewArraylist<storefile> ( This. Tempfiles.size ()); for(Path storefilepath:tempfiles) {Try{Storefiles.add (Hstore). This. Commitfile (Storefilepath, Cacheflushseqnum, status)); }Catch(IOException ex) {Log.error ("Failed to commit store file"+ Storefilepath, ex);//Try To delete the files we have committed before.           for(StoreFile sf:storefiles) {Path pathtodelete = Sf.getpath ();Try{Sf.deletereader (); }Catch(IOException DeleteEx) {Log.fatal ("Failed to delete store file we committed, halting"+ Pathtodelete, ex); Runtime.getruntime (). Halt (1); }          }Throw NewIOException ("Failed to commit the flush", ex); }      } for(StoreFile sf:storefiles) {if(Hstore. This. getcoprocessorhost ()! =NULL) {Hstore. This. Getcoprocessorhost (). Postflush (Hstore. This, SF);      } committedfiles.add (Sf.getpath ()); } hstore. This. flushedcellscount + = Cacheflushcount; Hstore. This. flushedcellssize + = Cacheflushsize;//ADD new file to store files. Clear snapshot too while we have the Store write lock.      returnHstore. This. Updatestorefiles (Storefiles, Snapshot.getid ()); }

This completes the operation of the flush of hbase.

Hregionserver Flush Operation Source Code Analysis