Giraph源碼分析 —— 啟動ZooKeeper服務

歡迎訪問： 西北工業大學 - 大資料與知識管理研究室 （Northwestern Polytechnical University - BigData and Knowledge Management Lab），連結：http://wowbigdata.cn/，http://wowbigdata.net.cn/，http://wowbigdata.com.cn。

說明：

(1) 實驗環境.

   三台伺服器：test165、test62、test63。test165同時是JobTracker和TaskTracker.

   測試例子：官網內建的SSSP程式，資料是自己類比產生。

   運行命令：hadoop jar giraph-examples-1.0.0-for-hadoop-0.20.203.0-jar-with-dependencies.jar org.apache.giraph.GiraphRunner org.apache.giraph.examples.SimpleShortestPathsVertex -vif org.apache.giraph.io.formats.JsonLongDoubleFloatDoubleVertexInputFormat -vip /user/giraph/SSSP -of org.apache.giraph.io.formats.IdWithValueTextOutputFormat -op /user/giraph/output-sssp-debug-7 -w 5

(2). 為節約空間，下文中所有代碼均為核心程式碼片段。

(3). core-site.xml中hadoop.tmp.dir的路徑設為：/home/hadoop/hadooptmp

(4).寫本文是多次調試完成的，故文中的JobID不一樣，讀者可理解為同一JobID.

(5). 後續文章也遵循上述規則。

1. org.apache.giraph.graph.GraphMapper類

 Giraph中自訂org.apache.giraph.graph.GraphMapper類來繼承Hadoop中的 org.apache.hadoop.mapreduce.Mapper<Object,Object,Object,Object>類，覆寫了setup()、map()、cleanup()和run()方法。GraphMapper類的說明如下：

This mapper that will execute the BSP graph tasks alloted to this worker. All tasks will be performed by calling the GraphTaskManager object managed by this GraphMapper wrapper classs. Since this mapper will not be passing data by key-value pairs through the MR framework, the Mapper parameter types are irrelevant, and set to Object type.

BSP的運算邏輯被封裝在GraphMapper類中，其擁有一GraphTaskManager對象，用來管理Job的tasks。每個GraphMapper對象都相當於BSP中的一個計算節點（compute node）。

在GraphMapper類中的setup()方法中，建立GraphTaskManager對象並調用其setup()方法進行一些初始化工作。如下：

  @Override  public void setup(Context context)    throws IOException, InterruptedException {    // Execute all Giraph-related role(s) assigned to this compute node.    // Roles can include "master," "worker," "zookeeper," or . . . ?    graphTaskManager = new GraphTaskManager<I, V, E, M>(context);    graphTaskManager.setup(      DistributedCache.getLocalCacheArchives(context.getConfiguration()));  }

map()方法為空白，因為所有操作都被封裝在了GraphTaskManager類中。在run()方法中調用GraphTaskManager對象的execute()方法進行BSP迭代計算。

@Override  public void run(Context context) throws IOException, InterruptedException {    // Notify the master quicker if there is worker failure rather than    // waiting for ZooKeeper to timeout and delete the ephemeral znodes    try {      setup(context);      while (context.nextKeyValue()) {        graphTaskManager.execute();      }      cleanup(context);      // Checkstyle exception due to needing to dump ZooKeeper failure    } catch (RuntimeException e) {      graphTaskManager.zooKeeperCleanup();      graphTaskManager.workerFailureCleanup();    }  }

2. org.apache.giraph.graph.GraphTaskManager 類

功能：The Giraph-specific business logic for a single BSP compute node in whatever underlying type of cluster our Giraph job will run on. Owning object will provide the glue into the underlying cluster framework and will call this object to perform Giraph work.

下面講述setup()方法，代碼如下。

 /**   * Called by owner of this GraphTaskManager on each compute node   * @param zkPathList the path to the ZK jars we need to run the job   */  public void setup(Path[] zkPathList) throws IOException, InterruptedException {    context.setStatus("setup: Initializing Zookeeper services.");    locateZookeeperClasspath(zkPathList);    serverPortList = conf.getZookeeperList();    if (serverPortList == null && startZooKeeperManager()) {      return; // ZK connect/startup failed    }    if (zkManager != null && zkManager.runsZooKeeper()) {        LOG.info("setup: Chosen to run ZooKeeper...");    }    context.setStatus("setup: Connected to Zookeeper service " +serverPortList);    this.graphFunctions = determineGraphFunctions(conf, zkManager);    instantiateBspService(serverPortList, sessionMsecTimeout);  }

依次介紹每個方法的功能：

    1) locateZookeeperClasspath(zkPathList)：找到ZK jar的本機複本，其路徑為：/home/hadoop/hadooptmp/mapred/local/taskTracker/root/jobcache/job_201403270456_0001/jars/job.jar ,用於啟動ZooKeeper服務。
    2) startZooKeeperManager()，初始化和配置ZooKeeperManager。定義如下，

 /**   * Instantiate and configure ZooKeeperManager for this job. This will   * result in a Giraph-owned Zookeeper instance, a connection to an   * existing quorum as specified in the job configuration, or task failure   * @return true if this task should terminate   */  private boolean startZooKeeperManager()    throws IOException, InterruptedException {    zkManager = new ZooKeeperManager(context, conf);    context.setStatus("setup: Setting up Zookeeper manager.");    zkManager.setup();    if (zkManager.computationDone()) {      done = true;      return true;    }    zkManager.onlineZooKeeperServers();    serverPortList = zkManager.getZooKeeperServerPortString();    return false;  }

org.apache.giraph.zk.ZooKeeperManager 類，功能：Manages the election of ZooKeeper servers, starting/stopping the services, etc. 

ZooKeeperManager類的setup()定義如下：

/**   * Create the candidate stamps and decide on the servers to start if   * you are partition 0.   */  public void setup() throws IOException, InterruptedException {    createCandidateStamp();    getZooKeeperServerList();  }

createCandidateStamp()方法在 HDFS上 的_bsp/_defaultZkManagerDir/job_201403301409_0006/_task 目錄下為每個task建立一個檔案，檔案內容為空白。檔案名稱為原生Hostname+taskPartition，如下：

運行時指定了5個workers(-w 5)，再加上一個master，所有上面有6個task。

getZooKeeperServerList()方法中，taskPartition為0的task會調用createZooKeeperServerList()方法建立ZooKeeper server List，也是建立一個空檔案，通過檔案名稱來描述Zookeeper servers。

createZooKeeperServerList核心代碼如下：

/**   * Task 0 will call this to create the ZooKeeper server list.  The result is   * a file that describes the ZooKeeper servers through the filename.   */  private void createZooKeeperServerList() throws IOException,      InterruptedException {    Map<String, Integer> hostnameTaskMap = Maps.newTreeMap();    while (true) {      FileStatus [] fileStatusArray = fs.listStatus(taskDirectory);      hostnameTaskMap.clear();      if (fileStatusArray.length > 0) {        for (FileStatus fileStatus : fileStatusArray) {            String[] hostnameTaskArray =              fileStatus.getPath().getName().split(HOSTNAME_TASK_SEPARATOR);             if (!hostnameTaskMap.containsKey(hostnameTaskArray[0])) {            hostnameTaskMap.put(hostnameTaskArray[0],                new Integer(hostnameTaskArray[1]));          }        }        if (hostnameTaskMap.size() >= serverCount) {          break;        }        Thread.sleep(pollMsecs);      }    }  }

    首先擷取taskDirectory（_bsp/_defaultZkManagerDir/job_201403301409_0006/_task）目錄下檔案，如果目前的目錄下有檔案，則把檔案名稱（Hostname+taskPartition）中的Hostname和taskPartition存入到hostNameTaskMap中。掃描taskDirectory目錄後，若hostNameTaskMap的size大於serverCount（等於GiraphConstants.java中的ZOOKEEPER_SERVER_COUNT變數，定義為1），就停止外層的迴圈。外層迴圈的目的是：因為taskDirectory下的檔案每個task檔案時多個task在分布式條件下建立的，有可能task 0在此建立server List時，別的task還沒有產生後task檔案。Giraph預設為每個Job啟動一個ZooKeeper服務，也就是說只有一個task會啟動ZooKeeper服務。

    經過多次測試，task 0總是被選為ZooKeeper Server ，因為在同一進程中，掃描taskDirectory時，只有它對應的task 檔案（其他task的檔案還沒有產生好），然後退出for迴圈，發現hostNameTaskMap的size等於1，直接退出while迴圈。那麼此處就選了test162 0。

    最後，建立了檔案：_bsp/_defaultZkManagerDir/job_201403301409_0006/zkServerList_test162 0

    onlineZooKeeperServers()，根據zkServerList_test162 0檔案，Task 0 先產生zoo.cfg設定檔，使用ProcessBuilder來建立ZooKeeper服務進程，然後Task 0 再通過socket串連到ZooKeeper服務進程上，最後建立檔案 _bsp/_defaultZkManagerDir/job_201403301409_0006/_zkServer/test162 0 來標記master任務已完成。worker一直在進行迴圈檢測master是否產生好 _bsp/_defaultZkManagerDir/job_201403301409_0006/_zkServer/test162 0，即worker等待直到master上的ZooKeeper服務已經啟動完成。

啟動ZooKeeper服務的命令如下：

    3) determineGraphFunctions()。

     GraphTaskManager類中有CentralizedServiceMaster對象和CentralizedServiceWorker 對象，分別對應於master和worker。每個BSP compute node扮演的角色判定邏輯如下：

     a) If not split master, everyone does the everything and/or running ZooKeeper. 

     b) If split master/worker, masters also run ZooKeeper

     c) If split master/worker == true and giraph.zkList is set, the master will not instantiate a ZK instance, but will assume a quorum is already active on the cluster for Giraph to use.

該判定在GraphTaskManager 類中的靜態方法determineGraphFunctions()中定義，片段代碼如下：

 private static GraphFunctions determineGraphFunctions(      ImmutableClassesGiraphConfiguration conf,      ZooKeeperManager zkManager) {    // What functions should this mapper do?    if (!splitMasterWorker) {      if ((zkManager != null) && zkManager.runsZooKeeper()) {        functions = GraphFunctions.ALL;      } else {        functions = GraphFunctions.ALL_EXCEPT_ZOOKEEPER;      }    } else {      if (zkAlreadyProvided) {        int masterCount = conf.getZooKeeperServerCount();        if (taskPartition < masterCount) {          functions = GraphFunctions.MASTER_ONLY;        } else {          functions = GraphFunctions.WORKER_ONLY;        }      } else {        if ((zkManager != null) && zkManager.runsZooKeeper()) {          functions = GraphFunctions.MASTER_ZOOKEEPER_ONLY;        } else {          functions = GraphFunctions.WORKER_ONLY;        }      }    }    return functions;  }

    預設的，Giraph會區分master和worker。會在master上面啟動zookeeper服務，不會在worker上啟動ZooKeeper服務。那麼Task 0 就是master+ZooKeeper，其他Tasks就是workers。