MAC OS X Yosemite installing Hadoop 2.6 Records

The entire installation process is divided into four parts:

I. Installation homebrew

Two. SSH localhost

Three. Installing Hadoop has configuration file settings (pseudo-distributed)

Four. Running Chestnuts

I. Installation homebrew

Installing Hadoop with homebrew is simple and easy, and it's a hassle to have previously installed Hadoop on Cygwin on Windows.

About homebrew can view its official website http://brew.sh/ can also view the homebrew next generation OS X Suite management tool article.

$ RUBY-E "$ (curl-fssl https://raw.githubusercontent.com/Homebrew/install/master/install)"

The previous line of commands is then copied to the terminal execution and the installation is complete.

Two. SSH localhost

Because installing Hadoop requires remote login, you need to install the SSH tool. Mac OS x only needs to be able to use SSH by ticking the "Remote Login" tab of "share" in "System Preferences".

SSH can see the principle and application of SSH article.

If the SSH public key has not been generated, use the command: (View ~/.SSH/ID_DSA and~/.SSH/ID_DSA.The pub does not exist to know whether the public key has been generated before, or directly execute SSH localhost to see if it succeeds)

$ ssh-keygen-t rsa-p "" $ cat $HOME/.ssh/id_rsa.pub >> $HOME/.ssh/authorized_keys

The following command is intended to store the public key in the other's public key folder, in order to avoid entering the password each time ssh localhost is logged on remotely.

Three. Install Hadoop because it is on one of its own computers, it uses pseudo-distributed mode.

<span style= "FONT-SIZE:18PX;" >$ Brew Install Hadoop</span>

Hadoop installed successfully, of course, the latest stable version of the installation, I installed the latest 2.6. Hadoop will be installed on a predetermined path.

Here is the modification of the configuration file:

hadoop-env.sh

Files in/usr/local/cellar/hadoop/2.6.0/libexec/etc/hadoop/hadoop-env.sh

Will

hadoop_opts= "$HADOOP _opts-djava.net.preferipv4stack=true"

modified to

export HADOOP_OPTS="$HADOOP_OPTS -Djava.net.preferIPv4Stack=true -Djava.security.krb5.realm= -Djava.security.krb5.kdc="

edit core-site.xml

/usr/local/cellar/hadoop/2.6.0/libexec/etc/hadoop/ Core-site.xml

<configuration>  <property>     <name>hadoop.tmp.dir</name>  <value>/usr/local/Cellar/hadoop/hdfs/tmp</value>    <description>A base for other temporary directories.</description>  </property>  <property>     <name>fs.default.name</name>                                          <value>hdfs://localhost:9000</value>                               </property>                                                        

EditMapred-site.xml may be the file name Mapred-site.xml.templete, change the name can be changed.

< Span style= "line-height:19px" > /usr/local/cellar/hadoop/2.6.0/libexec/etc/hadoop/ mapred-site.xml is just beginning to be blank file

<configuration>       <property>         <name>mapred.job.tracker</name>         <value>localhost:9010</value>       </property> </configuration>

EditHdfs-site.xml

/usr/local/cellar/hadoop/2.6.0/libexec/etc/hadoop/ Hdfs-site.xml

<configuration>    <property>      <name>dfs.replication</name>      <value>1</value>     </property> </configuration>

At this point, the configuration file is finished editing.

Next is the installation directory that goes into Hadoop /usr/local/cellar/hadoop/2.6.0/sbin

then execute./start-dfs.sh and./start-yarn.sh to start Hadoop. But there will be a warning:WARN util. nativecodeloader:unable to load Native-hadoop library for your platform ... using Builtin-java classes where applicable this pair There is no impact on the operation of Hadoop, so let's talk about this warning later.

To start Hadoop, avoid first entering the installation directory each time, and then executing the./start-dfs.sh and./start-yarn.sh so troublesome, so in the edit ~/.profiles file, add the following two lines:

alias hstart="/usr/local/Cellar/hadoop/2.6.0/sbin/start-dfs.sh;/usr/local/Cellar/hadoop/2.6.0/sbin/start-yarn.sh"alias hstop="/usr/local/Cellar/hadoop/2.6.0/sbin/stop-yarn.sh;/usr/local/Cellar/hadoop/2.6.0/sbin/stop-dfs.sh"

And then execute$ source ~/.profile update. This makes it possible to start Hadoop with Hstart and Hstop, which are both straightforward and straightforward.

But before you start Hadoop, you'll need to format HDFs

$ hadoop namenode -format

At this point, you can start Hadoop with Hstart. You can use the $JPS command to verify that Hadoop is running.

Four. Running Chestnuts

after the installation is over, you'll want to see if you can use it, and Hadoop comes with an example. $ Hadoop jar <path to the hadoop-examples file> pi

$ Hadoop Jar/usr/local/cellar/hadoop/2.3.0/libexec/share/hadoop/mapreduce/hadoop-mapreduce-examples-2.3.0.jar Pi 2 5

The resulting result may be this:

wrote input for map #0Wrote the input for map #1Starting Job ... Job finished in 1.685 secondsestimated value of Pi is 3.60000000000000000000

It can then be monitored from the web side.

Resource Manager: http://localhost:50070JobTracker: http://localhost:8088Specific Node Information: http://localhost:8042

They can access the HDFS filesystem, and can also get the result output file.

Note: The warning mentioned earlier is about the Hadoop local library, the Hadoop local library, which is designed to improve efficiency or some functional component libraries that cannot be implemented in Java. Currently only supports *unix, not supported on Mac OS X and Cygwin. Native Hadoop Library

but I read an article that says you can also support a local library in Mac OS X, and the workaround is to Hadoop uses local libraries to improve efficiency with MAC OSX

MAC OS X Yosemite installing Hadoop 2.6 Records