HBase Source code Parsing (ii) Hmaster main class member resolution

This article is based on HBase-0.94.1 analysis of the main class members of Hmaster.

Hmaster is the central node in the HBase master/slave cluster architecture. Typically, an hbase cluster has multiple hmaster nodes, one of which is active master and the rest is Backup master.

The main class members of Hmaster are as follows:

1.ZooKeeper Listening

These classes are inherited from Zookeeperlistener.

/******************************zookeeperlistener and zookeeperwatcher******************************************* **/  //Our ZK client.  Private Zookeeperwatcher ZooKeeper;  Manager and ZK listener for master election//inherit Zookeeperlistener  private Activemastermanager Activemastermanager ;  Region server tracker//inherits Zookeeperlistener  private Regionservertracker regionservertracker;  Draining region server tracker//inherits Zookeeperlistener  private Drainingservertracker drainingservertracker;/** Manager of assignment nodes in zookeeper*/  AssignmentManager AssignmentManager;

1.1 Zookeeperwatcher

Zookeeperwatcher is the only class in HBase that implements the Watcher interface, and other internal classes that need to perceive zookeeper are passed through Zookeeperwatcher.registerlistener ( Zookeeperlistener Listener) method to register with the Zookeeperwatcher instance.

Zookeeper is a Zookeeperwatcher object that is initialized in the Hmaster constructor; It is a single zookeeper watcher, and each hmaster,regionserver,client instantiates a zookeeperwatcher.

1.2 Zookeeperlistener

Zookeeperlistener is the base class that HBase uses to listen for zookeeper events.

(1) The zookeeperwatcher of a process executes a method of the Zookeeperlistener parent class. In order to receive events from watcher, each listener must register itself through Zookeeperwatcher.registerlistener.

(2) Zookeeperlistener subclasses need to rewrite their own methods of interest ( Note that Wather will be blocked when calling the listeners method, so the listener method does not long-running)

Activemastermanager

This class is used to handle all events related to the master election on the master side.

(1). Listen and respond to the ZK notification about master Znode (ZK notifications), including nodecreated and nodedeleted.

(2). Includes a blocking method: Hold backup masters and wait for active master to hang up.

(3). This class is initialized in Hmaster, Hmaster calls Blockuntilbecomingactivemaster () to block waiting to become active master.

Regionservertracker

This class is used to track all online status Regionserver through ZK

Drainingservertracker

This class is used to track all regionserver that are in the on-line/offline running state via ZK.

AssignmentManager

AssignmentManager is responsible for managing the distribution of the region.

2. RPC Server

Hmaster uses the hbase RPC mechanism to encapsulate itself as an RPC Server, providing RPC invocation services externally.

  RPC Server for the Hmaster  private final rpcserver rpcserver;  /**   * This servers address.   */  private final inetsocketaddress Isa;

3.HBase File System: Masterfilesystem

Masterfilesystem abstracts the series of interfaces required for Hmaster to interact with the lower-level file system

It is initialized in the Hmaster.finishinitialization () method.

  File System Manager for the master FS operations//abstracts the series of operations required by Hmaster to interact with the underlying file system  private Masterfilesystem Filesystemmanager

private void Finishinitialization (Monitoredtask status, Boolean Masterrecovery)  throws IOException, Interruptedexception, Keeperexception {//will be set to true for the active master identifier    isactivemaster = true;      /* We are already active master and start initializing the related components.        ...    This.masteractivetime = System.currenttimemillis ();    Todo:do this using Dependency injection, using Picocontainer, Guice or Spring.    1. Create the HBase file system Masterfilesystem. (Abstract a series of interfaces required for Hmaster to interact with low-level file systems)    This.filesystemmanager = new Masterfilesystem (this, this, metrics, masterrecovery);        ...}

4. Regionserver and Catalog Management

Servermanager

Servermanager manages all the regionserver information. In fact, Regionservertracker,drainingservertracker,assignmentmanager requires Servermanager as a constructor parameter when initializing.

  /******************************servermanager****************************************************/  /** Server Manager to deal with the region server info*/  private Servermanager Servermanager;

Regionservertracker,drainingservertracker,assignmentmanager is initialized at Initializezkbasedsystemtracker ().

  private void Initializezkbasedsystemtrackers () throws IOException,      interruptedexception, keeperexception {    This.catalogtracker = new Catalogtracker (This.zookeeper, this.conf, this);    This.catalogTracker.start ();    This.balancer = loadbalancerfactory.getloadbalancer (conf);    This.assignmentmanager = new AssignmentManager (this, Servermanager,        This.catalogtracker, This.balancer, This.executorservice);    Zookeeper.registerlistenerfirst (AssignmentManager);    This.regionservertracker = new Regionservertracker (ZooKeeper, this,        This.servermanager);    This.regionServerTracker.start ();    This.drainingservertracker = new Drainingservertracker (ZooKeeper, this,      This.servermanager);    This.drainingServerTracker.start ();

Catalogtracker

Catalogtracker tracking the availability of catalog tables (-root-and. META. Tables)

Clusterstatustracker

The Clusterstatustracker tracks the configuration of the cluster on the zookeeper (Tracker on cluster settings up in zookeeper). Unlike Clusterstatustracker and Clusterstatus, the latter is just a mirrored data structure that stores cluster current views. Clusterstatustracker is a property information that is used to track the configuration of a cluster on zookeeper.

  Manager of catalog Regions  private Catalogtracker Catalogtracker;  Cluster status ZK Tracker and local setter  private Clusterstatustracker clusterstatustracker;

5.HBase Actuator Service: Executorservice

Executorservice is a generic executor service class that abstracts a threadpool, a queue, and a runnable thread (handler role)

  /******************************executorservice****************************************************/  // Instance of the HBase executor service.  Executorservice Executorservice;

The variable is initialized in the Finishinitialization method.

  private void Finishinitialization (Monitoredtask status, Boolean Masterrecovery)  throws IOException, Interruptedexception, keeperexception {    ...    if (!masterrecovery) {      //Initialize Executorservice and Servermanager        this.executorservice = new Executorservice ( getServerName (). toString ());//Maintain a threadpool and queue      This.servermanager = new Servermanager (this);// Manage all Regionserver    }

6. Load Balancing: LoadBalancer

LoadBalancer's duty is to maintain the load balance between regionservers; Balancerchore threads do some balancer related cleanup work.

Users can customize their load balancing policies by implementing the LoadBalancer interface. By default, HBase uses the Org.apache.hadoop.hbase.master.DefaultLoadBalancer class for load balancing.

  /******************************loadbalancer, balancerchore****************************************/  Private LoadBalancer balancer;  Private Thread Balancerchore;

LoadBalancer is initialized by calling Loadbalancerfactory.getloadbalancer (conf) in the Initializezkbasedsystemtrackers () method.

  private void Initializezkbasedsystemtrackers () throws IOException,      interruptedexception, keeperexception {    This.balancer = loadbalancerfactory.getloadbalancer (conf); ...}

As can be seen from the Loadbalancerfactory code, LoadBalancer is specified by the parameter Hbase_master_loadbalancer_class (Hbase.master.loadbalancer.class), The default value is Defaultloadbalancer.class

public class Loadbalancerfactory {  /**   * Create a loadblanacer from the given Conf.   * @param conf   * @return A {@link LoadBalancer}   *  /public static LoadBalancer Getloadbalancer (Configuration conf) {    //Create the balancer    class<? extends loadbalancer> Balancerklass = Conf.getclass (        Hconstants.hbase_master_loadbalancer_class,        Defaultloadbalancer.class, loadbalancer.class);    Return Reflectionutils.newinstance (Balancerklass, conf);}  }

7. Table Descriptor Management: Tabledescriptor

The Tabledescriptor interface describes a series of operations that are used to manage table descriptors, and Fstabledescriptor is the implementation class for that interface.

  /******************************tabledescriptors***************************************************/  Private Tabledescriptors tabledescriptors;

Class in the Hmaster.finishinitialization method

private void Finishinitialization (Monitoredtask status, Boolean Masterrecovery)  throws IOException, Interruptedexception, keeperexception {    ....    2. Initialize the tabledescriptors and read the htable description from the file system.    This.tabledescriptors =      New Fstabledescriptors (This.fileSystemManager.getFileSystem (),   This.fileSystemManager.getRootDir ());

The Ftabledescriptor constructor is as follows: Fstabledescriptors can read and change all table descriptors according to FS and RootDir

  Public fstabledescriptors (Final FileSystem FS, final Path RootDir,      final Boolean fsreadonly) {    super ();    This.fs = FS;    This.rootdir = RootDir;    This.fsreadonly = fsreadonly;  }

To Fstabledescriptor. Fstabledescriptor gets RootDir The following path (these paths are obtained through Fileutils.gettabledirs (Fs,rootdir) and filtered). Use the name of path (also known as the file name) as the TableName.

  @Override public  map<string, htabledescriptor> getAll ()  throws IOException {    map<string, htabledescriptor> Htds = new treemap<string, htabledescriptor> ();    Get all tablename    list<path> tabledirs = Fsutils.gettabledirs (FS, rootdir);    for (Path d:tabledirs) {     htabledescriptor htd = null;   try {       htd = Get (D.getname ());   } catch (FileNotFoundException Fnfe) {   } if (HTD = = null)     continue;
   
    htds.put (D.getname (), HTD);  } return Htds; }
   

8. Htable Image Management: Snapshotmanager

Snapshotmanager manages the snapshots generation and loading (taking and restoring) process.

  Monitor for snapshot of hbase tables  private Snapshotmanager Snapshotmanager;  The image manager of HBase tables

9. Collaborative processor management: Coprocessorprotocol Map

  /******************************protocolhandlers Management coprocessor Protocol Registration *********************/  //Registered Master protocol Handlers  private classtoinstancemap<coprocessorprotocol>      protocolhandlers = Mutableclasstoinstancemap.create ();

10. Health Check and garbage cleanup: various chore threads

  /** the health check chore. * *  private healthcheckchore healthcheckchore;//Health Check  /****************************** garbage collection: Catalogjanitor, logcleaner,hfilecleaner*********************/  private catalogjanitor catalogjanitorchore;  Timed Scan-meta-, garbage collection for useless region  Logcleaner private Logcleaner;  Private Hfilecleaner Hfilecleaner;

11. Other

Todo

HBase Source code Parsing (ii) Hmaster main class member resolution