Install, configure Hadoop based on Mac OS OS

Install Hadoop on your Mac

For those of me who have never been in touch with *nix before, it is a waste of time to use the command line to do a series of things. Close up this record to do the backup.

Get Java

My Mac is running OS X 10.7 Lion, and Java has been installed before, and you can use the Java-version command in the utility---terminal to confirm the Java version. If you do not have Java installed, you can also go to the following URL download: http://support.apple.com/kb/dl1421.

Get Hadoop

The specific address of their own Baidu bar. I downloaded the stable version of 1.0.4.

After download, unzip, I place the directory is/users/billy/hadoop.

Setting environment variables

Before you actually start Hadoop, there are three files that need to be configured.

But before we do, we need to set up several Windows-like environment variables to make it easier to knock commands at the command line later.

Export Hadoop_home=/users/billy/hadoop

Export path= $PATH: $HADOOP _home/bin

Configure hadoop-env.sh

Under the hadoop->conf directory, locate hadoop-env.sh, and open the edit to set up the following:

Export Java_home=/library/java/home (Remove comments)

Export hadoop_heapsize=2000 (Remove comments)

Export hadoop_opts= "-djava.security.krb5.realm=ox. ac.uk-djava.security.krb5.kdc=kdc0.ox.ac.uk:kdc1.ox.ac.uk "(Remove comments)

Note that the third configuration is best configured on OS X, otherwise it will be reported as "unable to load realms info from Scdynamicstore".

Configure Core-site.xml <?xml version= "1.0"?>
<?xml-stylesheet type= "text/xsl" href= "configuration.xsl"?><configuration>

<property>

<name>hadoop.tmp.dir</name>

<value>/users/billy/hadoop/tmp/hadoop-${user.name}</value>

<description>a base for other temporary directories.</description>

</property>

<property>

<name>fs.default.name</name>

<value>hdfs://localhost:8020</value>

</property>

</configuration>

Configuring Hdfs-site.xml<?xml version= "1.0"?>
<?xml-stylesheet type= "text/xsl" href= "configuration.xsl"?>

<configuration>
<property>
<name>dfs.replication</name>
<value>1</value>
</property>

</configuration>

Configure Mapred-site.xml

<configuration>

<property>
<name>mapred.job.tracker</name>
<value>localhost:8021</value>
</property>

<property>
<name>mapred.tasktracker.map.tasks.maximum</name>
<value>2</value>
</property>

<property>
<name>mapred.tasktracker.reduce.tasks.maximum</name>
<value>2</value>
</property>
</configuration>

Installing HDFs

After the above configuration, the installation of HDFs can be done.

$HADOOP _home/bin/hadoop name Node-format

If it goes well, you will get output similar to the following:

Billymatomacbook-air:hadoop billy$ $HADOOP _home/bin/hadoop Namenode-format

Warning: $HADOOP _home is deprecated.

12/12/02 17:11:12 INFO Namenode. Namenode:startup_msg:

/************************************************************

Startup_msg:starting NameNode

Startup_msg:host = billymatomacbook-air.local/192.168.1.102

Startup_msg:args = [-format]

Startup_msg:version = 1.0.4

Startup_msg:build = Https://svn.apache.org/repos/asf/hadoop/common/branches/branch-1.0-r 1393290; Compiled by ' Hortonfo ' on Wed Oct 3 05:13:58 UTC 2012

************************************************************/

12/12/02 17:11:12 INFO util. GSET:VM type = 64-bit

12/12/02 17:11:12 INFO util. gset:2% max memory = 39.9175 MB

12/12/02 17:11:12 INFO util. gset:capacity = 2^22 = 4194304 entries

12/12/02 17:11:12 INFO util. gset:recommended=4194304, actual=4194304

12/12/02 17:11:12 INFO Namenode. Fsnamesystem:fsowner=billy

12/12/02 17:11:12 INFO Namenode. Fsnamesystem:supergroup=supergroup

12/12/02 17:11:12 INFO Namenode. Fsnamesystem:ispermissionenabled=true

12/12/02 17:11:12 INFO Namenode. fsnamesystem:dfs.block.invalidate.limit=100

12/12/02 17:11:12 INFO Namenode. Fsnamesystem:isaccesstokenenabled=false accesskeyupdateinterval=0 min (s), Accesstokenlifetime=0 min (s)

12/12/02 17:11:13 INFO Namenode. namenode:caching file names occuring more than times

12/12/02 17:11:13 INFO Common. Storage:image file of size 111 saved in 0 seconds.

12/12/02 17:11:13 INFO Common. Storage:storage Directory/users/billy/hadoop/tmp/hadoop-billy/dfs/name has been successfully formatted.

12/12/02 17:11:13 INFO Namenode. Namenode:shutdown_msg:

/************************************************************

Shutdown_msg:shutting down NameNode at billymatomacbook-air.local/192.168.1.102

************************************************************/

Start Hadoop

It's simple, a command is done.

$HADOOP _home/bin/start-all.sh

If it goes well, you will usually be asked to enter the password for three times.

Simple commissioning

If you want to try it out successfully, you can test it with your own example:

$hadoop jar $HADOOP _home/hadoop-example-1.0.4.jar Pi 10 100

If successful, there will be similar results:

Billymatomacbook-air:hadoop billy$ Hadoop jar $HADOOP _home/hadoop-examples-1.0.4.jar Pi 10 100

Warning: $HADOOP _home is deprecated.

Number of Maps = 10

Samples per Map = 100

Wrote input for Map #0

Wrote input for Map #1

Wrote input for Map #2

Wrote input for Map #3

Wrote input for Map #4

Wrote input for Map #5

Wrote input for Map #6

Wrote input for Map #7

Wrote input for Map #8

Wrote input for Map #9

Here, a single-node Hadoop installation is done on Mac OS X. Go to your Hadoop world!

Install, configure Hadoop based on Mac OS OS