Easily build hadoop-1.2.1 pseudo-distributions

Easily build hadoop-1.2.1 pseudo-distributions

Take CentOS for example:

CentOS Virtual machine Installation: http://blog.csdn.net/baolibin528/article/details/32918565

Network settings: http://blog.csdn.net/baolibin528/article/details/43797107

Pietty Usage: http://blog.csdn.net/baolibin528/article/details/43822509

WINSCP Usage: http://blog.csdn.net/baolibin528/article/details/43819289

As long as the virtual system is installed, everything can be remotely operated by the software.

1. Set IP

Configuring a pseudo-distribution runs on only one machine, so using localhost instead of an IP address is also possible.

If you want to connect to Linux with Remote Tools (Pietty, WINSCP, and so on), you must set the IP address.

When the system is installed, the IP address can be well-equipped.

If a command is available: Service network restart//restart Network

Ifconfig//View IP information

[Root@baolibin ~]# ifconfig
eth0      linkencap:ethernet  HWaddr 00:0c:29:44:a3:a5
          inetaddr : 192.168.1.100 bcast:192.168.1.255 mask:255.255.255.0
          inet6 addr:fe80::20c:29ff:fe44:a3a5/64 scope:link
          up Broadcast runningmulticast  mtu:1500  metric:1
          RX packets:97040errors:0 dropped:0 overruns:0 frame:0
          TX packets:10935errors:0 dropped:0 overruns:0 carrier:0
          collisions:0txqueuelen:1000
          RX bytes:141600363 ( 135.0 MiB)  TX bytes:1033124 (1008.9KiB)
 
lo        Link encap:localloopback
          inetaddr:127.0.0.1  mask:255.0.0.0
          Inet6 Addr::: 1/128scope:host up loopbackrunning mtu:16436 metric:1
          RX Packets:16 errors:0 dropped:0 overruns:0frame:0
          TX packets:16errors:0 dropped:0 overruns:0 carrier:0
          collisions:0txqueuelen:0
          RX bytes:1184 (1.1KiB)  TX bytes:1184 (1.1 KiB)

2. Turn off the firewall

Execute command: Service iptables stop

Validation: Service iptables Status

Once the above command is executed, the firewall will open again when it is booted.

Permanently shut down the firewall: chkconfig iptables off

Verification: Chkconfig--list | grep iptables

[root@baolibin~]# chkconfig--list | grep iptables
iptables        0: Off  1: Off 2: Off  3: Off  4: off 5: Off  6: Off

3. Set host Name:

Command 1:hostname host Name

This command temporarily modifies the host name and is not valid after a reboot. The following command is a permanent modification of the host name.

Command 2:vim/etc/sysconfig/network

Networking=yes
hostname=baolibin
gateway=192.168.1.1

4, IP and hostname binding:

Command: vim/etc/hosts

127.0.0.1   localhost localhost.localdomain localhost4localhost4.localdomain4
:: 1         localhost Localhost.localdomainlocalhost6 localhost6.localdomain6
192.168.1.100  baolibin

5, set up SSH password-free login:

Execute command: ssh-keygen-t rsa-p "

Cat ~/.ssh/id_rsa.pub >>~/.ssh/authorized_keys

chmod 600~/.ssh/authorized_keys

[root@baolibin~]# ssh 192.168.1.100 last
login:sunfeb-21:01:30 from Baolibin
[root@baolibin~]#

6. Install the JDK:

Give executable permission to JDK installation: Chmodu+x Jdk-6u45-linux-x64.bin

Unzip the JDK:./jdk-6u45-linux-x64.bin

Rename the unpacked folder: MV JDK1.6.0_45JDK

Modify permissions: Chown-r Hadoop:hadoop JDK

In the/etc/profile file, add:

#set Java Environment
export JAVA_HOME=/USR/LOCAL/JDK export
jre_home=/usr/local/jdk/jre
Export Classpath=.: $JAVA _home/lib: $JAVA _home/jre/lib
export path= $JAVA _home/bin: $JAVA _home/jre/bin: $PATH

[root@baolibinlocal]# vim/etc/profile
[root@baolibinlocal]# source/etc/profile
[root@baolibinlocal]# java -version
java Version "1.6.0_45"
Java (tm) Seruntime environment (build 1.6.0_45-b06)
Javahotspot (tm) 64- Bit Server VM (build 20.45-b01, Mixed mode)
[root@baolibinlocal]#

7. Install Hadoop:

Decompression HADOOP:TAR-ZXVF hadoop-1.2.1.tar.gz

Modify permissions: Chown-r Hadoop:hadoop hadoop-1.2.1

Renaming: MV Hadoop-1.2.1hadoop

Editor: Vim/etc/profile

   #sethadoop Environment
Export hadoop_home=/usr/hadoop
export path= $PATH: $HADOOP _home/bin

Source/etc/profile

Modify configuration files under the Conf directory hadoop-env.sh, Core-site.xml, Hdfs-site.xml, Mapred-site.xml

Hadoop-env.sh:

Export JAVA_HOME=/USR/LOCAL/JDK

Core-site.xml:

<configuration>
   <property>
           <name>fs.default.name</name>
           <value>hdfs ://192.168.1.100:9000</value>
  </property>
  <property>
           <name>hadoop.tmp.dir </name>
           <value>/home/hadoop/dfs</value>
   </property>
</configuration>

Note: The previous write <value>/usr/hadoop/tmp</value> each use needs reformatting, otherwise jobtracker will automatically disappear after a while, change to non-TMP directory is normal.

Hdfs-site.xml:

<configuration>
 <property>
       <name>dfs.replication</name>
      <value>1< /value>
  </property>
  <property>
      <name>dfs.permissions</name>
      < value>false</value>
 </property>
</configuration>

Mapred-site.xml:

<configuration>
 <property>
       <name>mapred.job.tracker</name>
       <value> 192.168.1.100:9001</value>
 </property>
</configuration>

Modify VIM masters and vim slaves under Conf

Content is 192.168.1.100

8. Format:

Execute command: Hadoop namenode-format

[Hadoop@baolibin ~]$ JPS 29363 JPS [Hadoop@baolibin ~]$ cd/usr/hadoop/bin [hadoop@baolibin bin]$ Hadoop Namenode-format
 
Warning: $HADOOP _home is deprecated. 15/02/15 21:04:06 INFO Namenode.
Namenode:startup_msg:/************************************************************ startup_msg:starting NameNode   Startup_msg:host = baolibin/192.168.1.100 Startup_msg:args = [-format] startup_msg:version = 1.2.1 STARTUP_MSG: Build =https://svn.apache.org/repos/asf/hadoop/common/branches/branch-1.2-r 1503152;compiled by ' Mattf ' on Mon Jul 22 1 5:23:09 PDT Startup_msg:java = 1.6.0_45 ************************************************************/15/02/15 21 : 04:06 INFO util. Gset:computingcapacity for map blocksmap 15/02/15 21:04:06 INFO util. Gset:vmtype = 64-bit 15/02/15 21:04:06 INFO util. gset:2.0% maxmemory = 1013645312 15/02/15 21:04:06 INFO util. gset:capacity = 2^21 = 2097152 entries 15/02/15 21:04:06 INFO util. gset:recommended=2097152, actual=2097152 15/02/15 21:04:07 infonamenode.fsnamesystem:fsowner=hadoop 15/02/15 21:04:07 infonamenode.fsnamesystem:supergroup=supergroup 15/02/15 21:04:07 infonamenode.fsnamesystem:ispermissionenabled=false 15/02/15 21:04:07 INFOnamenode.FSNamesystem: dfs.block.invalidate.limit=100 15/02/15 21:04:07 Infonamenode.fsnamesystem:isaccesstokenenabled=false Accesskeyupdateinterval=0min (s), Accesstokenlifetime=0 min (s) 15/02/15 21:04:07 INFO Namenode. FSEditLog:dfs.namenode.edits.toleration.length = 0 15/02/15 21:04:07 INFO namenode. namenode:caching file names occuring more than times 15/02/15 21:04:08 INFO Common.
Storage:image file/usr/hadoop/tmp/dfs/name/current/fsimage of size bytes saved in0 seconds. 15/02/15 21:04:08 INFO Namenode. Fseditlog:closing edit log:position=4, editlog=/usr/hadoop/tmp/dfs/name/current/edits 15/02/15 21:04:08 INFO Namenode . Fseditlog:close success:truncate to 4, editlog=/usr/hadoop/tmp/dfs/name/current/edits 15/02/15 21:04:08 INFO Common. Storage:storage Directory/usr/hadooP/tmp/dfs/name has been successfully formatted. 15/02/15 21:04:08 INFO Namenode. Namenode:shutdown_msg:/************************************************************ shutdown_msg:shutting down NameNode atbaolibin/192.168.1.100 ************************************************************/[Hadoop@baolibin bin]$

9. Start Hadoop:

Execute command: start-all.sh

[hadoop@baolibinbin]$ start-all.sh
Warning: $HADOOP _home is deprecated.
 
Startingnamenode, logging to/usr/hadoop/libexec/. /logs/hadoop-hadoop-namenode-baolibin.out
192.168.1.100:starting Datanode, logging to/usr/hadoop/libexec/. /logs/hadoop-hadoop-datanode-baolibin.out
192.168.1.100:starting Secondarynamenode, logging to/usr/hadoop/ libexec/. /logs/hadoop-hadoop-secondarynamenode-baolibin.out
Startingjobtracker, logging to/usr/hadoop/libexec/. /logs/hadoop-hadoop-jobtracker-baolibin.out
192.168.1.100:starting tasktracker, logging to/usr/hadoop/libexec /.. /logs/hadoop-hadoop-tasktracker-baolibin.out
[hadoop@baolibinbin]$

To view a process:

[hadoop@baolibinbin]$ JPS
29707SecondaryNameNode
29804JobTracker
29928TaskTracker
29585DataNode
30049 Jps
29470NameNode
[hadoop@baolibinbin]$

10, in the browser view: http://192.168.1.100:50070 http://192.168.1.100:50030

Log in with remote software and open it directly on the Windows Web page:

50070:

50030:

11. Turn off Hadoop

Execute command: stop-all.sh

[hadoop@baolibinbin]$ stop-all.sh
Warning: $HADOOP _home is deprecated.
 
Stoppingjobtracker
192.168.1.100:stopping tasktracker
stoppingnamenode
192.168.1.100:stopping Datanode
192.168.1.100:stopping secondarynamenode
[hadoop@baolibinbin]$

12. There is no possible reason for Namenode at startup:

(1) No formatting

(2) environment variable set error

(3) IP and hostname bind failed