Configuring Hadoop pseudo-Distribution mode

The Linux Mint 64bit,hadoop uses version 1.2.1.


1, set up SSH


installation of SSH-related packages:


sudo apt install openssh-client openssh-server then use one of the following two commands to start/close sshd:


sudo/etc/init.d/ssh start|stop


sudo service ssh start|stop if successfully launched SSHD, we can see the following similar results:


$ ps-e | grep ssh


2766? Unlimited Ssh-agent


10558? Unlimited sshd This time, if you run the following SSH log on this machine command, you will be prompted to enter the password:


ssh localhost now all we have to do is make it do not need to enter a password:


$ ssh-keygen-t rsa-p ' F ~/.ssh/id_rsa #一个空密码的SSH密钥


$ cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys so that's all right. If it still does not work, the key file may be improperly set.


2, configure Hadoop


will extract hadoop-1.2.1 to ~/and create directory hadoop-env under ~/, and continue to create the following directory structure under hadoop-env:


├──dfs


│├──checkpoint1


│├──data1


│├──data2


│└──name1


└──test


└──input configuration file Hadoop-1.2.1/conf/core-site.xml:


<?xml version= "1.0"?>


<?xml-stylesheet type= "text/xsl" href= configuration.xsl "?>"


<!--put site-specific property overrides in this file. -->


<configuration>


<property>


<name>fs.default.name</name>


<value>hdfs://localhost:9000</value>


<final>true</final>


</property>


</configuration>fs.default.name to specify the URI of the HDFs, if the port is not provided in value, the default is 8020.


configuration file Hadoop-1.2.1/conf/hdfs-site.xml:


<?xml version= "1.0"?>


<?xml-stylesheet type= "text/xsl" href= configuration.xsl "?>"


<!--put site-specific property overrides in this file. -->


<configuration>


<property>


<name>dfs.name.dir</name>


<value>~/hadoop-env/dfs/name1</value>


<final>true</final>


</property>


<property>


<name>dfs.data.dir</name>


<value>~/hadoop-env/dfs/data1,~/hadoop-env/dfs/data2</value>


<final>true</final>


</property>


<property>


<name>fs.checkpoint.dir</name>


<value>~/hadoop-env/dfs/checkpoint1</value>


<final>true</final>


</property>


</configuration> Dfs.name.dir Specify the directory where Namenode stores metadata, you can specify multiple directories, which are separated by commas; Dfs.data.dir Specify the directory where the data is stored datanode, you can specify multiple directories, and Fs.checkpoint.dir specify the directory where the secondary Namenode stores checkpoints.


configuration file Hadoop-1.2.1/conf/mapred-site.xml:


<?xml version= "1.0"?>


<?xml-stylesheet type= "text/xsl" href= configuration.xsl "?>"


<!--put site-specific property overrides in this file. -->


<configuration>


<property>


<name>mapred.job.tracker</name>


<value>localhost:9001</value>


<final>true</final>


</property>


</configuration>3, Test


First format HDFs:


./hadoop-1.2.1/bin/hadoop namenode-format start HDFs and MapReduce processes:


$/hadoop-1.2.1/bin/start-dfs.sh


$./hadoop-1.2.1/bin/start-mapred.sh when an error is being started, such as Localhost:Error:JAVA_HOME is not set., you need to be in./hadoop-1.2.1/conf/ hadoop-env.sh file, export java_home, for example:


export java_home=~/jdk1.7.0_25 How to check for successful startup:


the first method is to use the JPS command (Java Virtual Machine Process Status Tool), which should have the following similar output:


$ JPS


13592 DataNode


13728 Secondarynamenode


13837 Jobtracker


12864 Namenode


13955 Tasktracker


16069 JPS The second method is the browser login http://localhost:50030 view jobtracker,http://localhost:50070 See Namenode. If you are using the flip-wall software, you may be able to make an error when you visit, the simplest way is to turn off the flip-wall software.

Another way to
is to view log files.


Now we create two files in the ~/hadoop-env/test/input directory:


$ echo "Hello World" > Test1.txt


$ echo "Hi,world" > Test2.txt import these two files into HDFs:


./hadoop-1.2.1/bin/hadoop dfs-put hadoop-env/test/input/output/View:


$./hadoop-1.2.1/bin/hadoop Dfs-ls/


Found 2 Items


drwxr-xr-x-user supergroup 0 2013-10-22 22:07/test


drwxr-xr-x-user supergroup 0 2013-10-22 21:58/tmp


$/hadoop-1.2.1/bin/hadoop dfs-ls/test


Found 2 Items


-rw-r--r--3 user supergroup 2013-10-22 22:07/test/test1.txt


-rw-r--r--3 user supergroup 9 2013-10-22 22:07/test/test2.txt complete.