JStorm與Storm源碼分析（四）--均衡調度器，EvenScheduler

標籤：type   signed   ret   div   二維   list   判斷   rto   介面   

EvenScheduler同DefaultScheduler一樣,同樣實現了IScheduler介面, 
由下面代碼可以看出:

(ns backtype.storm.scheduler.EvenScheduler  (:use [backtype.storm util log config])  (:require [clojure.set :as set])  (:import [backtype.storm.scheduler IScheduler Topologies            Cluster TopologyDetails WorkerSlot ExecutorDetails])  (:gen-class    :implements [backtype.storm.scheduler.IScheduler]))EvenScheduler是一個對資源進行均勻分配的調度器：(defn -prepare [this conf]  )(defn -schedule [this ^Topologies topologies ^Cluster cluster]  (schedule-topologies-evenly topologies cluster))

它是通過調用schedule-topologies-evenly方法來完成任務分配的. 
schedule-topologies-evenly方法的具體定義如下:

(defn schedule-topologies-evenly [^Topologies topologies ^Cluster cluster]  ;;通過調用cluster對象的needsSchedulingTopologies方法來擷取所有需要進行任務調度的Topology集合,  ;;needsSchedulingTopologies方法具體定義如fn1所示.  ;;判斷Topology是否需要進行任務調度的依據在fn2中有說明.  (let [needs-scheduling-topologies (.needsSchedulingTopologies cluster topologies)]       (doseq [^TopologyDetails topology needs-scheduling-topologies        ;;對需要進行任務調度的Topology中的每一個,首先擷取它的topology-id,        :let [topology-id (.getId topology)          ;;調用schedule-topology方法擷取計算得到的<executor,node+port>類型集合new-assignment          ;;schedule-topology方法具體定義如fn3所示.          new-assignment (schedule-topology topology cluster)          ;;將new-assignment的鍵和值顛倒擷取<node+port,executors>集合.          node+port->executors (reverse-map new-assignment)]]       ;;對於前面擷取的<node+port,executors>集合中的每一項進行以下操作.      (doseq [[node+port executors] node+port->executors    ;;用node和port資訊構造WorkerSlot對象,並將其作為slot          :let [^WorkerSlot slot (WorkerSlot. (first node+port) (last node+port))            ;;下面兩行代碼:對於executors集合中的每一項,構造ExecutorDetail對象,            ;;並返回一個ExecutorDetails集合作為executors            executors (for [[start-task end-task] executors]               (ExecutorDetails. start-task end-task))]]         ;;調用cluster的assign方法將計算出來的slot分配給與該Topology相對應的executors        (.assign cluster slot topology-id executors)))))

fn1:

/** * 擷取所有需要調度的Topology,並以集合的形式返回 */public List<TopologyDetails> needsSchedulingTopologies(Topologies topologies) {    List<TopologyDetails> ret = new ArrayList<TopologyDetails>();    for (TopologyDetails topology : topologies.getTopologies()) {        if (needsScheduling(topology)) {            ret.add(topology);        }    }    return ret;}

fn2:

/** * 判斷Topology是否需要進行任務調度的依據有兩個: * 1.Topology設定的NumWorkers數目是否大於已經分配給Topology的Worker數目 * 2.該Topology尚未分配的Executor的數目是否大於0 */public boolean needsScheduling(TopologyDetails topology) {    int desiredNumWorkers = topology.getNumWorkers();    int assignedNumWorkers = this.getAssignedNumWorkers(topology);    if (desiredNumWorkers > assignedNumWorkers) {        return true;    }    return this.getUnassignedExecutors(topology).size() > 0;}

fn3:

;;該方法會根據叢集當前的可用資源對Topology進行任務分配(defn- schedule-topology [^TopologyDetails topology ^Cluster cluster]  ;;擷取topology-id  (let [topology-id (.getId topology)        ;;調用cluster的getAvailableSlots方法擷取叢集當前可用的slot資源,        ;;將其轉換為<node,port>集合并賦值給available-slots        ;;getAvailableSlots主要負責計算當前叢集中還沒有使用的Supervisor連接埠        available-slots (->> (.getAvailableSlots cluster)              (map #(vector (.getNodeId %) (.getPort %))))        ;;調用getExecutors擷取Topology的所有Executor資訊,        ;;將其轉換為<start-task-id,end-task-id>集合,        ;;然後賦值給all-executors並返回        all-executors (->> topology              .getExecutors              (map #(vector (.getStartTask %) (.getEndTask %)))              set)        ;;調用get-alive-assigned-node+port->executors方法(具體定義如fn3_1)        ;;計算當前該Topology已經分得的資源情況,        ;;最後返回一個<node+port,executors>集合并將其賦值給變數alive-assigned        ;;參數為cluster資訊和topology-id        alive-assigned (get-alive-assigned-node+port->executors cluster topology-id)        ;;計算當前Topology可以使用的slot數目,並將其賦予total-slots-to-use,        ;;該值的具體內容為下面兩個值的最小值:        ;;1.Topology中設定的Worker數目        ;;2.當前available-slots加上alive-assigned數目        total-slots-to-use (min (.getNumWorkers topology)               (+ (count available-slots) (count alive-assigned)))        ;;對available-slots進行排序,計算需要分配的slot數目(total-slots-to-use減去alive-assigned)        ;;最後從排序後的available-slots集合中按順序去除這些slot並賦值給reassign-slots        reassign-slots (take (- total-slots-to-use (count alive-assigned))             (sort-slots available-slots))        ;;通過比較all-executors跟已經分配的Executor集合間的差異,擷取需要進行分配的Executor集合        reassign-executors (sort (set/difference all-executors (set (apply concat (vals alive-assigned)))))        ;;將上述計算得到的reassign-executors與reassign-slots進行關聯,轉換為<executor,slot>映射集合,        ;;並賦值給reassignment,此時有兩種情況:        ;;1.reassign-executors數目少於reassign-slots數目:意味著當前叢集中的可用資源比較多,        ;;eg.reassign-executors為(e1,e2,e3),reassign-slots為(s1,s2,s3,s4,s5),        ;;那麼匹配結果為{[e1,s1],[e2,s2],[e3,s3]}        ;;2.reassign-executors數目多於reassign-slots數目:意味著當前叢集的可用資源非常有限,        ;;eg.reassign-executors為(e1,e2,e3,e4,e5,e6),reassign-slots為(s1,s2),        ;;此時會有多個Executor被分配到同一個slot上,返回的結果可能是:        ;;{[e1,s1],[e2,s1],[e3,s2],[e4,s1],[e5,s2],[e6,s2]}        reassignment (into {}           (map vector                reassign-executors                ;; for some reason it goes into infinite loop without limiting the repeat-seq                (repeat-seq (count reassign-executors) reassign-slots)))]    ;;判斷reassignment是否為空白,若不為空白則列印內容為可用的slot資訊的日誌    (when-not (empty? reassignment)      (log-message "Available slots: " (pr-str available-slots))      )    ;;返回計算得到類型為<executor,[node,port]>的集合reassignment，    reassignment))

fn3_1:

;;該方法用於擷取Topology當前已經分配得到的資源(defn get-alive-assigned-node+port->executors [cluster topology-id]  ;;調用cluster的getAssignmentById擷取該Topology當前的assignment  (let [existing-assignment (.getAssignmentById cluster topology-id)        ;;判斷當前的assignment是否為空白,若不為空白,則擷取其中的<executor,slot>資訊        executor->slot (if existing-assignment                         (.getExecutorToSlot existing-assignment)                         {})         ;;將前面擷取到的<executor,slot>轉換為<executor,[node+port]>集合        executor->node+port (into {} (for [[^ExecutorDetails executor ^WorkerSlot slot] executor->slot           :let [executor [(.getStartTask executor) (.getEndTask executor)]                 node+port [(.getNodeId slot) (.getPort slot)]]]       {executor node+port}))    ;;將前面的<executor,[node+port]>集合轉換為<[node+port],executors>集合        alive-assigned (reverse-map executor->node+port)]    ;;返回得到的<[node+port],executors>集合    alive-assigned))

註：學習李明老師等Storm源碼分析和陳敏敏老師等Storm技術內幕與大資料實踐的筆記整理。 
歡迎關注下面二維碼進行技術交流： 

JStorm與Storm源碼分析（四）--均衡調度器，EvenScheduler