MAC OS X Yosemite installing Hadoop 2.6 Records

The entire installation process is divided into four parts:

I. Installation homebrew

Two. SSH localhost

Three. Installing Hadoop has configuration file settings (pseudo-distributed)

Four. Execution of chestnuts

I. Installation homebrew

Using homebrew to install Hadoop is simple and convenient, and before installing Hadoop on Windows Cygwin, it feels good to be in trouble.

About homebrew can view its official website http://brew.sh/ also be able to view homebrew next generation OS X Suite management tools article.

$ RUBY-E "$ (curl-fssl https://raw.githubusercontent.com/Homebrew/install/master/install)"

Then copy the above line of command to the terminal to run and install it.

Two. SSH localhost

Since installing Hadoop requires remote login, you need to install the SSH tool. Mac OS x only needs to be able to use SSH in the "Sharing" tab of "System Preferences" in "Remote login".

Many other SSH can see the SSH principle and application article.

Assuming that the SSH public key has not been generated, use the command: (View ~/.SSH/ID_DSA and~/.SSH/ID_DSA.The pub does not exist to know whether the public key has been generated before, or directly run SSH localhost to see if it will succeed)

$ ssh-keygen-t rsa-p "" $ cat $HOME/.ssh/id_rsa.pub >> $HOME/.ssh/authorized_keys

The latter command is intended to store the public key in the other's public key storage directory, in order to avoid entering password every time the ssh localhost is run remotely.

Three. Install Hadoop because it is on one of its own computers, it uses pseudo-distributed mode.

<span style= "FONT-SIZE:18PX;" >$ Brew Install Hadoop</span>

Hadoop installed successfully, of course, the latest stable version of this installation, I installed the latest 2.6. Hadoop will be installed on a predetermined path.

Here are the changes to the configuration file:

hadoop-env.sh

Files in/usr/local/cellar/hadoop/2.6.0/libexec/etc/hadoop/hadoop-env.sh

Will

hadoop_opts= "$HADOOP _opts-djava.net.preferipv4stack=true"

changed to

export HADOOP_OPTS="$HADOOP_OPTS -Djava.net.preferIPv4Stack=true -Djava.security.krb5.realm= -Djava.security.krb5.kdc="

edit core-site.xml

/usr/local/cellar/hadoop/2.6.0/libexec/etc/hadoop/ Core-site.xml

<configuration>  <property>     <name>hadoop.tmp.dir</name>  <value>/usr/local/Cellar/hadoop/hdfs/tmp</value>    <description>A base for other temporary directories.</description>  </property>  <property>     <name>fs.default.name</name>                                          <value>hdfs://localhost:9000</value>                               </property>                                                        

Note: Fs.default.name saves the location of the Namenode, and the HDFs and MapReduce components need it, which is why it is in the Core-site.xml file instead of the Hdfs-site.xml file.

Edit Mapred-site.xml may be a filename called mapred-site.xml.templete, can not change the name.

/usr/local/cellar/hadoop/2.6.0/libexec/etc/hadoop/ Mapred-site.xml just started out as a blank file.

<configuration>       <property>         <name>mapred.job.tracker</name>         <value>localhost:9010</value>       </property> </configuration>

The variable Mapred.job.tracker holds the location of the Jobtracker, because only the MapReduce component needs to know the location, so it's in the Mapred-site.xml file today.

Edit Hdfs-site.xml

/usr/local/cellar/hadoop/2.6.0/libexec/etc/hadoop/ Hdfs-site.xml

<configuration>    <property>      <name>dfs.replication</name>      <value>1</value>     </property> </configuration>

The variable dfs.replication specifies the number of replications per HDFs database. Typically 3, because we only have one host and one pseudo-distributed pattern of datanode, change this value to 1.

At this point, profile editing is complete.

The next step is to go to the installation folder in Hadoop. /usr/local/cellar/hadoop/2.6.0/sbin

then run./start-dfs.sh and./start-yarn.sh to start Hadoop. Just a warning will appear here:WARN util. nativecodeloader:unable to load Native-hadoop library for your platform ... using Builtin-java classes where applicable this pair The implementation of Hadoop has no effect, and again, after this warning.

In order to start Hadoop, avoid each time you first go to the installation folder, and then run./start-dfs.sh and./start-yarn.sh so troublesome, so in the edit ~/.profiles file, plus for example the following two lines:

alias hstart="/usr/local/Cellar/hadoop/2.6.0/sbin/start-dfs.sh;/usr/local/Cellar/hadoop/2.6.0/sbin/start-yarn.sh"alias hstop="/usr/local/Cellar/hadoop/2.6.0/sbin/stop-yarn.sh;/usr/local/Cellar/hadoop/2.6.0/sbin/stop-dfs.sh"

And then run$ source ~/.profile update. This makes it possible to start Hadoop with Hstart and Hstop, both simple and straightforward.

Just format HDFs before you start Hadoop

$ hadoop namenode -format

Then you can start Hadoop with Hstart. The ability to use the JPS command to verify that Hadoop is executing.

Four. Execution of chestnuts

after the installation is done, you'll want to see if you can use it, and Hadoop comes with a sample. $ Hadoop jar <path to the hadoop-examples file> pi

$ Hadoop Jar/usr/local/cellar/hadoop/2.3.0/libexec/share/hadoop/mapreduce/hadoop-mapreduce-examples-2.3.0.jar Pi 2 5

The resulting result may be this:

wrote input for map #0Wrote the input for map #1Starting Job ... Job finished in 1.685 secondsestimated value of Pi is 3.60000000000000000000

It can then be monitored from the web side.

Resource Manager: http://localhost:50070JobTracker: http://localhost:8088Specific Node Information: http://localhost:8042

The resulting output file can also be obtained through their access to the HDFS filesystem.

Note: The warning mentioned earlier is about the Hadoop local library, the Hadoop local library, which is designed to improve efficiency or some functional component libraries that cannot be implemented in Java. Support for *unix is not supported on Mac OS X and Cygwin at the moment. Native Hadoop Library

just I saw an article saying that it is also possible to support a local library in Mac OS X, and the workaround is Hadoop uses local libraries to improve efficiency with MAC OSX

MAC OS X Yosemite installing Hadoop 2.6 Records