Install hadoop-2.8 process under Mac or Linux

Installing Hadoop on your Mac

Java under the 1,mac

Ll/usr/bin |grep ' Java '

Default:

/system/library/frameworks/javavm.framework/versions/current/commands/java

Specific installation path:

/library/java/javavirtualmachines/below,

Java_home is/library/java/javavirtualmachines/jdk1.7.0_75/contents/home.

SSH under the 2,mac

First Test ssh localhost,

If it does not succeed, see whether to start allow RemoteLogin, and then try again if it fails then continue to the following

Ssh-keygen-t Dsa-p "-F ~/.SSH/ID_DSA

Ssh-keygen indicates the creation of a secret key;

-T denotes the secret key type;-P is used to provide a secret word;-f Specifies the generated key file.

This command creates two files Id_dsa and id_dsa.pub in the "~/.ssh/" folder, which is a pair of SSH private and public keys.

Next, append the public key to the authorized key and enter:

Cat~/.ssh/id_dsa.pub >> ~/.ssh/authorized_keys

Try again

SSH localhost

If this cannot be opened, the workaround is as follows:

Enter your Mac's system Preferences –> share –> check remote login and set allow access for all users.

3, download the image, mirror, select a tar.gz

http://mirror.bit.edu.cn/apache/hadoop/common/

4, setting environment variables

VI ~/.bash_profile or/etc/profile

Exportjava_home=/library/java/javavirtualmachines/jdk1.7.0_75/contents/home

Exportjre_home= $JAVA _home/jre

Exporthadoop_home=/users/xiaolong.zhao/ittools/hadoop

#是防止出现: Warning: Warning error $HADOOP _home is deprecated.

Exporthadoop_home_warn_suppress=1

Exportpath= $JAVA _home/bin: $JRE _home/bin: $ hadoop_home/sbin: $HADOOP _home/bin: $PATH

Note: The only difference between Linux and Mac installation is that there will be a default JDK path under the Mac, and the SSH configuration

5, Configuration hadoop-env.sh

Under the/users/xiaolong.zhao/ittools/hadoop/etc/hadoop directory, locate hadoop-env.sh, and open the edit to set up the following:

Export Java_home=${java_home} (remove comments)

Export hadoop_heapsize=2000 (Remove comments)

Export hadoop_opts= "-djava.security.krb5.realm=ox. AC. uk-djava.security.krb5.kdc=kdc0.ox.ac.uk:kdc1.ox.ac.uk "(Remove comments)

6, 4 core file configurations

6.1. Configuration Core-site.xml--(DEFAULTFS) specifies the hostname and port of the Namenode

<?xmlversion= "1.0"  encoding= "UTF-8"?>  <configuration>      <!-- fs.defaultFS -  This is a URI that describes the Namenode node in the cluster (including protocol, host name, port number), and each machine in the cluster needs to know the Namenode address.      datanode nodes are first registered on the Namenode so that their data can be used. The standalone client program interacts with Datanode through this URI to get a list of blocks of files. -->    <property>         <name >fs.defaultfs</name>        <value>hdfs://localhost :8000</value>      </property>         <!-- hadoop.tmp.dir  is the underlying configuration that Hadoop file system relies on, and many paths depend on it.       if Namenode and Datanode are not configured in the Hdfs-site.xml, the default is placed in this path-->     <property>         <name>hadoop.tmp.dir</ name>        <value>/users/time/documents/hadoop-2.7.3/tmp</value>           <description>A base for othertemporary directories.</description>      </property> </configuration>

6.2. Configuration Hdfs-site.xml--Specifies the number of default parameter replicas for HDFs

 <?xmlversion= "1.0"  encoding= "UTF-8"?>  <configuration>        <!-- dfs.replication -It determines the number of data backups of the file blocks in the   system. For a real application, it should be set to 3.       less than three backups may affect the reliability of the data-->        < Property>          <name>dfs.replication</name >           <value>1</value>         </property>          <!--the physical storage of  datanaode and Namenode, if not set, its value hadoop.tmp.dir-->         <property>            <name> dfs.namenode.name.dir</name>            < value>/home/hdfs/name</value>        </property>          <!--  can not be set, Namenode web  monitoring   port, default is 0.0.0.0:50070-->         <property>             <name>dfs.namenode.http-address</name>              <value>localhost:57000</value>         </property>         <property>           <name>dfs.datanode.data.dir</name>           <value>/home/hdfs/data</value>         </property> </configuration>

6.3 Mapred-site.xml Configuration

<property> <name>mapreduce.framework.name</name> <value>yarn</value> </property>

6.4 Yarn-site.xml Configuration

<?xmlversion= "1.0"?>  <configuration>    <!--  resourcemanager -->   <!-- client visit address -->   <property>   <name>yarn.resourcemanager.address</name>  < value>localhost:18040</value>   </property>    <!--  Applicationmaster visit address-->  <property>   <name> Yarn.resourcemanager.scheduler.address</name>  <value>localhost:18030</value>    </property>    <!-- nodemanager visit address  -->  <property>   <name> yarn.resourcemanager.resource-tracker.address</name>  <value> localhost:18025</ value>   </property>    <!-- weB visit address -->  <property>   <name> yarn.resourcemanager.webapp.address</name>  <value>localhost:18088</value>    </property>    <!-- nodemanager support mapreduce  service-->   <property>   <name> yarn.nodemanager.aux-services</name>  <value>mapreduce_shuffle</value>    </property>  </configuration>

7, formatted Namenode

HDFs Namenode-format

8. Start Hadoop

8.1sbin/start-all.sh

8.2 Web Test

8.2.1 Namenode-web

http://localhost:50070

8.2.2resourcemanager-web

http://localhost:18088

8.2.3 Nodemanager-web

http://localhost:8042

This article is from the "Dream in the Cloud" blog, please be sure to keep this source http://computerdragon.blog.51cto.com/6235984/1939523

Install hadoop-2.8 process under Mac or Linux