YARN & HDFS2 Installation and configuration Kerberos

Today, trying to configure Kerberos on the Hadoop 2.x development cluster, I've encountered some problems and recorded

Set up Hadoop security

Core-site.xml

<property>  
   <name>hadoop.security.authentication</name>  
   <value>kerberos</ value>  
</property>  
<property>  
   <name>hadoop.security.authorization</name >  
   <value>true</value>  
</property>

Hadoop.security.authentication default is simple, that is, based on the Linux operating system authentication method, the client calls the WhoAmI command, and then RPC call to the server, It is easy for a malicious user to forge an identical user in another host. Here we change to Kerberos.

Set HDFs security

Hdfs-site.xml

<property> <name>dfs.block.access.token.enable</name> &LT;VALUE&GT;TRUE&LT;/VALUE&G  
T </property> <property> <name>dfs.https.enable</name> <value>false</ value> </property> <property> <name>dfs.namenode.https-address</name> & lt;value>dev80.hadoop:50470</value> </property> <property> &LT;NAME&GT;DFS.HTTPS.PORT&L t;/name> <value>50470</value> </property> <property> <name>dfs.na menode.keytab.file</name> <value>/etc/hadoop.keytab</value> </property> <property > <name>dfs.namenode.kerberos.principal</name> &LT;VALUE&GT;HADOOP/_HOST@DIANPING.COM&L t;/value> </property> <property> &LT;NAME&GT;DFS.NAMENODE.KERBEROS.HTTPS.PRINCIPAL&LT;/NAME&G  
        T <value>host/_host@dianping.com</value> </property> <property> <name>dfs.namenode.sec ondary.http-address</name> <value>dev80.hadoop:50090</value> </property> <propert y> <name>dfs.namenode.secondary.https-port</name> <value>50470</value> &L t;/property> <property> <name>dfs.namenode.secondary.keytab.file</name> <val ue>/etc/hadoop.keytab</value> </property> <property> <name>dfs.namenode.secondary .kerberos.principal</name> <value>hadoop/_HOST@DIANPING.COM</value> </property> < Property> <name>dfs.namenode.secondary.kerberos.https.principal</name> <value>hos t/_host@dianping.com</value> </property> <property> <name>dfs.datanode.data.dir.perm  
        </name><value>700</value> </property> <property> <name>dfs.datanode.address</name&  
        Gt <value>0.0.0.0:1003</value> </property> <property> <name>dfs.datanode.http.add ress</name> <value>0.0.0.0:1007</value> </property> <property> <n Ame>dfs.datanode.https.address</name> <value>0.0.0.0:1005</value> </property> ;p roperty> <name>dfs.datanode.keytab.file</name> <value>/etc/hadoop.keytab</va  
        lue> </property> <property> <name>dfs.datanode.kerberos.principal</name> <value>hadoop/_HOST@DIANPING.COM</value> </property> <property> <name>dfs.da Tanode.kerberos.https.principal</name> <value>host/_HOST@DIANPING.COM</value> </property&  
Gt <Property> <name>dfs.datanode.data.dir.perm</name> <value>700</value> < /property> <property> <name>dfs.datanode.address</name> <value>0.0.0.0:10  
        03</value> </property> <property> <name>dfs.datanode.http.address</name> <value>0.0.0.0:1007</value> </property> <property> <name>dfs.datanode.htt  
        ps.address</name> <value>0.0.0.0:1005</value> </property> <property> <name>dfs.datanode.keytab.file</name> <value>/etc/hadoop.keytab</value> </property > <property> <name>dfs.datanode.kerberos.principal</name> <value>hadoop/_ host@dianping.com</value> </property> <property> &LT;NAME&GT;DFS.DATANODE.KERBEROS.HTTPS.P  
Rincipal</name>        <value>host/_HOST@DIANPING.COM</value> </property> <property> &LT;NAME&GT;DF S.web.authentication.kerberos.principal</name> <value>HTTP/_HOST@DIANPING.COM</value> </p roperty> <property> <name>dfs.web.authentication.kerberos.keytab</name> &LT;VALUE&G T;/etc/hadoop.keytab</value> <description> the Kerberos keytab file with the CRE  
      Dentials for the HTTP Kerberos principal used by Hadoop-auth in the HTTP endpoint. </description> </property>

There are several points to note in the configuration of 1. Dfs.datanode.address represents the hostname or IP address to which the data transceiver RPC server is bound, and if security is turned on, the port number must be less than 1024 (privileged port). Otherwise, start Datanode will report "Cannot start secure cluster without privileged resources" error