Mac OSX System Brew install Hadoop Installation Guide

Mac OSX System Brew install Hadoop Installation Guide

1. Brew Install Hadoop
2. Configure Core-site.xml: Configure the HDFs file address (remember to chmod the corresponding folder, otherwise it will not start HDFs properly) and Namenode RPC traffic port
3. Configuring the map reduce communication port in Mapred-site.xml
4. Configures the number of Datanode replicas in Hdfs-site.xml.
5. Format HDFs File system
6. Start Hadoop and yarn Namenode datanode mapred, which is: Run start_all.sh
7. (optional) Create the address of the current user in HDFs: Hadoop fs-mkdir-p/user/fan
8. Start, the Web UI address defaults to: http://localhost:50070.

problems encountered:One, as in Hdfs-put ~/file Fileonhdfs, appears error:warn org.apache.hadoop.hdfs.server.datanode.DataNode:Invalid directory in DF S.data.dir reasons are:

1. The file system set in Core-site.xml is full and needs to be reset and-format again.
2. The HDFs file system that is set in core-site.xml needs to be in the same folder as chmod 777

Two, as in Hadoop fs-ls. Command appears: ls: '. ': No such file or directory '

1. Please perform:hadoop fs -mkdir -p /user/[current login user]

configuration file Demo:

Configuring Core-site.xml, Hdfs-site.xml, Mapred-site.xml

* Core-site.xml:

XML code

1. <configuration>
3. <property>
4. <name>hadoop.tmp.dir</name>
5. <value>/usr/local/tmp/hadoop/hadoop-${user.name}</value>
6. <description>a base for other temporary directories.</description>
7. </property>
9. <property>
10. <name>fs.default.name</name>
11. <value>hdfs://localhost:8020</value>
12. </property>
14. </configuration>

* Hdfs-site.xml:

XML code

1. <configuration>
3. <property>
4. <name>dfs.replication</name>
5. <value>1</value>
6. </property>
8. </configuration>

* Mapred-site.xml:

XML code

1. <configuration>
3. <property>
4. <name>mapred.job.tracker</name>
5. <value>localhost:8021</value>
6. </property>
8. <property>
10. </configuration>

In fact, these are the basic configuration, online a lot. You can refer to these 3 addresses for a description of the configured values:

Wrote http://hadoop.apache.org/common/docs/r0.20.2/core-default.html
Http://hadoop.apache.org/common/docs/r0.20.2/hdfs-default.html
Http://hadoop.apache.org/common/docs/r0.20.0/mapred-default.html

2.3) format Hadoop file system HDFs. Execute command Hadoop Namenode-format

Mac OSX System Brew install Hadoop Installation Guide