The Learning prelude to Hadoop (i)--Installing and configuring Hadoop on Linux

Preface

I am now in the development of Android, but also in an information security company. The company's main is to do in the virtual machine running products, and the cloud computing, thought is to go to the cloud computing. Self-feeling mobile internet + cloud Computing + information security will be a very good direction, so join in.

Because it's Java origin. So it's natural to choose Hadoop.

Author System Environment

Linux:centos Release 6.5 (Final)
Jdk:java Version "1.7.0_75"
OpenJDK Runtime Environment (rhel-2.5.4.0.el6_6-x86_64 u75-b13)
OpenJDK 64-bit Server VM (build 24.75-b04, Mixed mode)
SSH:OPENSSH_5.3P1, OpenSSL 1.0.1e-fips 2013

hadoop:hadoop-1.2.1

Environment Construction1. Installing the JDK

About the installation of the JDK. This blog is not the scope of the discussion, we can self-Baidu or Google to install.

2. Configure SSH-free password login

If your machine can be networked, for example, the following operations

(1) # yum install Openssh-server # SSH installed

(2) # yum-y Install openssh-clients

# assuming SSH is not recognized after you install SSH, you need to install additional openssh-clients

(3) # Mkdir-p ~/.ssh # Assume that after you install SSH, these folders are not actively generated by yourself, please create your own

(4) # ssh-keygen-t Dsa-p "-F ~/.SSH/ID_DSA

Ssh-keygen indicates that the key is generated

-T means the specified generated key type

DSA is the meaning of DSA key authentication, that is, the key type

-P provides a passphrase

-f Specifies the generated key file

(5) # cat ~/.ssh/id_dsa.pub >> ~/.ssh/authorized_keys

# Add the public key to the public key file used for authentication. Authorized_keys is a public key file for authentication

(6) # Ssh-version

# Verify that SSH is installed and the correct installation will have similar results such as the following:

(7) At this point the main configuration is almost the same, but it is still not able to do password login. The final step is to configure the permissions properties for two things:

$ chmod ~/.ssh
$ chmod ~/.ssh/authorized_keys

These two steps are necessary. Otherwise, you may not be able to password login.

(8) $ ssh localhost # authentication

3. Install and run Hadoop

(1) Download hadoop-x.x.x

Unzip to the specified folder. Like/home/u.

(2) configuration information for changing configuration files

# vim ~/hadoop-1.2.1/conf/core-site.xml

# vim ~/hadoop-1.2.1/conf/hdfs-site.xml

# vim ~/hadoop-1.2.1/conf/mapred-site.xml

(3) # ~/hadoop-1.2.1/bin/hadoop Namenode-format # Format File system

(4) # ~/hadoop-1.2.1/bin/start-all.sh # start All Processes

(5) Verify that Hadoop is installed successfully

Enter the browser to verify by entering the following URL:

http://localhost:50030 (Web page for MapReduce)

http://localhost:50070 (HDFS Web page)

Validation Demo Sample:

Web page for MapReduce

Web pages for HDFs

problems encountered1. When starting Hadoop, always say Java_home is not configured

When I run bin/start-all.sh in the Hadoop folder using the shell command in the tutorial, I always report java_home is not set.

But I also set the java_home in the shell, and I went to check the system java_home is OK, such as the following:

This is very strange to me. If I find a forum, the situation inside is similar to mine. Just one of his words made me realize that I had a wrong place. That is the java_home of the above hint is actually to be set in the configuration file.

Run vim ~/hadoop-1.2.1/conf/hadoop-env.sh, this folder according to your own path to write, changes such as the following:

2.-bash:bin/hadoop:permission denied

If you are downloading directly from the network under Linux, there should be nothing. Just, suppose you are the same as the author. is to use WINSCP upload to Linux, there will be a small place to change.

Otherwise the error will be reported:

Being able to see this error is an error in the runtime file times we run Hadoop. Then we just have to change the permissions on this file to be able to. Since some of the other executable files will be used later, I have made a change to all the files (of course, because we are in the learning and testing phase, in order to avoid trouble, steal a lazy. Suppose you want to think about it from a security standpoint, we can't do that here .

3.Hadoop Safemode:on-hdfs Unavailable

Of course we've got some of the previous configuration information. There may also be a problem, that is, we are unable to access the Web page of HDFs.

This problem is actually a legacy of some of our previous misconfiguration. We talked about a permission problem running a file, and when we manipulate the format HDFs, this is exactly the problem of this privilege interfering. If I stop the process that was started before.

Then once again format is OK.

Information such as the following when formatting HDFs is correct:

The Learning prelude to Hadoop (i)--Installing and configuring Hadoop on Linux