hadoop官方文檔學習筆記（1）——resource manager HA

標籤：hadoop   ha   resource manager   

resource manager HA是hadoop自從2.4之後推出的功能，以Active/Standby的方式提供冗餘，目的是為了消除單點失敗的風險。

1、總體架構：

650) this.width=650;" src="http://hadoop.apache.org/docs/stable/hadoop-yarn/hadoop-yarn-site/images/rm-ha-overview.png" alt="Overview of ResourceManager High Availability" />

2、故障切換：有自動和手動兩種形式。

手動：如果以手動形式切換，使用yarn haadmin命令首先將Active節點轉為standby，再將standby節點轉為active。

自動：RM有基於zookeeper的節點選舉機制決定哪一個是活動節點。不需要像HDFS一樣部署一個zkfc守護進程，因為RM內嵌了這樣的功能。

做了rm的HA之後，所有節點和用戶端都要列出所有RM節點，串連時會用輪詢的方式遍曆，直到找到一個active的節點。如果活動節點down了，它們會繼續輪詢。這一動作被實現為org.apache.hadoop.yarn.client.ConfiguredRMFailoverProxyProvider類。可以通過重新實現該類，並在yarn.client.failover-proxy-provider 配置項中指定新的類名來改寫這一行為邏輯。

3、配置

Configuration Properties	Description
yarn.resourcemanager.zk-address	Address of the ZK-quorum. Used both for the state-store and embedded leader-election.
yarn.resourcemanager.ha.enabled	Enable RM HA.
yarn.resourcemanager.ha.rm-ids	List of logical IDs for the RMs. e.g., “rm1,rm2”.
yarn.resourcemanager.hostname.rm-id	For each rm-id, specify the hostname the RM corresponds to. Alternately, one could set each of the RM’s service addresses.
yarn.resourcemanager.address.rm-id	For each rm-id, specify host:port for clients to submit jobs. If set, overrides the hostname set in yarn.resourcemanager.hostname.rm-id.
yarn.resourcemanager.scheduler.address.rm-id	For each rm-id, specify scheduler host:port for ApplicationMasters to obtain resources. If set, overrides the hostname set in yarn.resourcemanager.hostname.rm-id.
yarn.resourcemanager.resource-tracker.address.rm-id	For each rm-id, specify host:port for NodeManagers to connect. If set, overrides the hostname set in yarn.resourcemanager.hostname.rm-id.
yarn.resourcemanager.admin.address.rm-id	For each rm-id, specify host:port for administrative commands. If set, overrides the hostname set in yarn.resourcemanager.hostname.rm-id.
yarn.resourcemanager.webapp.address.rm-id	For each rm-id, specify host:port of the RM web application corresponds to. You do not need this if you set yarn.http.policy to HTTPS_ONLY. If set, overrides the hostname set in yarn.resourcemanager.hostname.rm-id.
yarn.resourcemanager.webapp.https.address.rm-id	For each rm-id, specify host:port of the RM https web application corresponds to. You do not need this if you set yarn.http.policy to HTTP_ONLY. If set, overrides the hostname set in yarn.resourcemanager.hostname.rm-id.
yarn.resourcemanager.ha.id	Identifies the RM in the ensemble. This is optional; however, if set, admins have to ensure that all the RMs have their own IDs in the config.
yarn.resourcemanager.ha.automatic-failover.enabled	Enable automatic failover; By default, it is enabled only when HA is enabled.
yarn.resourcemanager.ha.automatic-failover.embedded	Use embedded leader-elector to pick the Active RM, when automatic failover is enabled. By default, it is enabled only when HA is enabled.
yarn.resourcemanager.cluster-id	Identifies the cluster. Used by the elector to ensure an RM doesn’t take over as Active for another cluster.
yarn.client.failover-proxy-provider	The class to be used by Clients, AMs and NMs to failover to the Active RM.
yarn.client.failover-max-attempts	The max number of times FailoverProxyProvider should attempt failover.
yarn.client.failover-sleep-base-ms	The sleep base (in milliseconds) to be used for calculating the exponential delay between failovers.
yarn.client.failover-sleep-max-ms	The maximum sleep time (in milliseconds) between failovers.
yarn.client.failover-retries	The number of retries per attempt to connect to a ResourceManager.
yarn.client.failover-retries-on-socket-timeouts	The number of retries per attempt to connect to a ResourceManager on socket timeouts.

4、樣本（最小配置）

<property>
  <name>yarn.resourcemanager.ha.enabled</name>
  <value>true</value>
</property>
<property>
  <name>yarn.resourcemanager.cluster-id</name>
  <value>cluster1</value>
</property>
<property>
  <name>yarn.resourcemanager.ha.rm-ids</name>
  <value>rm1,rm2</value>
</property>
<property>
  <name>yarn.resourcemanager.hostname.rm1</name>
  <value>master1</value>
</property>
<property>
  <name>yarn.resourcemanager.hostname.rm2</name>
  <value>master2</value>
</property>
<property>
  <name>yarn.resourcemanager.webapp.address.rm1</name>
  <value>master1:8088</value>
</property>
<property>
  <name>yarn.resourcemanager.webapp.address.rm2</name>
  <value>master2:8088</value>
</property>
<property>
  <name>yarn.resourcemanager.zk-address</name>
  <value>zk1:2181,zk2:2181,zk3:2181</value>
</property>

5、管理命令

查看節點狀態：
 $ yarn rmadmin -getServiceState rm1
 active

 $ yarn rmadmin -getServiceState rm2
 standby

故障切換：

 $ yarn rmadmin -transitionToStandby rm1
 $ yarn rmadmin -transitionToActive rm2

注意：
  開啟自動故障切換後，系統為防止造成腦裂或其它不一致的狀態，會拒絕人為管理HA狀態。如果非常清楚自己的行為，可以在切換命令中指定-forcemanual選項。

本文出自 “大資料學習探索” 部落格，請務必保留此出處http://bigdata1024.blog.51cto.com/6098731/1869156

hadoop官方文檔學習筆記（1）——resource manager HA