Reproduced Hadoop and Hive stand-alone environment setup

Reprinted from http://blog.csdn.net/yfkiss/article/details/7715476 and http://blog.csdn.net/yfkiss/article/details/7721329

Download Hadoop
Hadoop:
Http://www.apache.org/dyn/closer.cgi/hadoop/core/
The version downloaded here is 1.0.3
$ mkdir Hadoop
$ wget http://www.fayea.com/apache-mirror/hadoop/core/stable/hadoop-1.0.3.tar.gz.

Installing Java
First Use "java-version" to see if it is installed, and if the display is similar to Java version "1.7.0_147-icedtea means that it is installed
No installation can be consulted: http://blog.csdn.net/yang_hui1986527/article/details/6677450
Installation must be set Java_home and Classpath
My configuration:
Export Path=${path}:/usr/lib/jvm/java-6-openjdk-amd64/bin
Export java_home=/usr/lib/jvm/java-6-openjdk-amd64/
Export JRE_HOME=${JAVA_HOME}/JRE
Export Classpath=.:${jre_home}/lib:${java_home}/lib/dt.jar:${java_home}/lib/tools.jar
And put these two lines of statements in: ~/.BASHRC

Installing SSH
$ sudo apt-get install SSH
After the installation is complete, you need to confirm that you can log in to localhost with ssh password
$ ssh localhost
If you need a password to log in, you need to set:
$ ssh-keygen-t Dsa-p "-F ~/.SSH/ID_DSA
$ cat ~/.ssh/id_dsa.pub >> ~/.ssh/authorized_keys

Installing rsync
Rsync is the software for remote synchronization of Linux
$ sudo apt-get install rsync

Configure startup Hadoop
Extract:
$ TAR-ZXVF hadoop-1.0.3.tar.gz
Set Java_home
Edit conf/hadoop-env.sh file to find:
# Export Java_home=/usr/lib/j2sdk1.5-sun
Modified to:
Export java_home=/usr/lib/jvm/java-6-openjdk-amd64/

To modify a configuration file:
If you do not know where Java is, please use Whereis Java query
Modify Conf/core-site.xml:

[Plain]View Plaincopy

1. <configuration>
2. <property>
3. <name>fs.default.name</name>
4. <value>hdfs://localhost:9000</value>
5. </property>
7. </configuration>

Modify Conf/hdfs-site.xml:

[Plain]View Plaincopy

1. <configuration>
2. <property>
3. <name>dfs.replication</name>
4. <value>1</value>
5. </property>
6. <property>
7. <name>hadoop.tmp.dir</name>
8. <value>/home/work/hadoop_tmp</value>
9. </property>
10. </configuration>

Modify Conf/mapred-site.xml:

[Plain]View Plaincopy

1. <configuration>
2. <property>
3. <name>mapred.job.tracker</name>
4. <value>localhost:9001</value>
5. </property>
6. </configuration>

Initialize Hadoop Namenode:
$ bin/hadoop Namenode–format
Start:
$ bin/start-all.sh
Confirm Startup:
$ JPS
5146 Jps
4538 Tasktracker
4312 Jobtracker
4015 DataNode
4228 Secondarynamenode
3789 NameNode
Indicates a successful start

The following content is written to ~/.BASHRC:

[Plain]View Plaincopy

2. Alias hadoop= '/home/zxm/hadoop/hadoop-1.0.3/bin/hadoop '
4. Alias hls= ' Hadoop fs-ls '
6. Alias hlsr= ' Hadoop FS-LSR '
8. Alias hcp= ' Hadoop fs-cp '
10. Alias hmv= ' Hadoop fs-mv '
12. Alias hget= ' Hadoop fs-get '
14. Alias hput= ' Hadoop fs-put '
16. Alias hrm= ' Hadoop fs-rm '
18. Alias hmkdir= ' Hadoop fs-mkdir '
20. Alias hcat= ' Hadoop fs-cat '
22. Alias hrmr= ' Hadoop fs-rmr '
24. Alias hstat= ' Hadoop fs-stat '
26. Alias htest= ' Hadoop fs-test '
28. Alias htext= ' Hadoop fs-text '
30. Alias htouchz= ' Hadoop fs-touchz '
32. Alias hdu= ' Hadoop fs-du '
34. Alias hdus= ' Hadoop fs-dus '
36. Alias hchmod= ' Hadoop fs-chmod '
38. Alias hchgrp= ' Hadoop fs-chgrp '
40. Alias hchown= ' Hadoop fs-chown '
42. Alias htail= ' Hadoop fs-tail ' <span style= "font-family:arial, Helvetica, Sans-serif;" ><span style= "White-space:normal;" >
44. </span></span>

Solutions for common problems:
Issue 1: Running the HADOOP command appears "Warning: $HADOOP _home is deprecated." Alarm
FIX: Add export hadoop_home_warn_suppress=true to hadoop-env.sh

Problem 2:namenode cannot start
FIX: Delete/tmp/hadoop* execute Bin/hadoop namenode–format

After the Hadoop stand-alone environment has been successfully built, hive can be built.

To build a directory on HDFs:

[Plain]View Plaincopy

1. $ Hadoop fs-mkdir/tmp
2. $ Hadoop Fs-mkdir/user/hive/warehouse

Add Permissions:

[Plain]View Plaincopy

1. $ Hadoop fs-chmod g+w/tmp
2. $ Hadoop fs-chmod G+w/user/hive/warehouse

Download Extract hive:
$ wget http://labs.mop.com/apache-mirror/hive/stable/hive-0.8.1.tar.gz.
$ TAR-ZXVF hive-0.8.1.tar.gz

Set Hadoop_home, Hive_home, and add it to ~/.BASHRC

[Plain]View Plaincopy

1. Export hadoop_home=/home/zxm/hadoop/hadoop-1.0.3
2. Export hive_home=/home/work/hadoop/hive-0.8.1

Multi-user support
(Verify that MySQL is installed.)
Start MySQL:
$ mysql-u Root-p
Mysql>grant all on hive.* to [e-mail protected] identified by ' 123456 '

Modify the Hive Conf/hive-site.xml as follows:
[Plain]View Plaincopy

2. <property>
4. <name>javax.jdo.option.ConnectionURL</name>
6. <value>jdbc:mysql://localhost:3306/hive?createDatabaseIfNotExist=true<alue>
8. <DESCRIPTION>JDBC connect string for a JDBC metastore</description>
10. </property>
12. <property>
14. <name>javax.jdo.option.ConnectionDriverName</name>
16. <value>com.mysql.jdbc.Driver<alue>
18. <description>driver class name for a JDBC metastore</description>
20. </property>
22. <property>
24. <name>javax.jdo.option.ConnectionUserName</name>
26. <value>hive<alue>
28. <description>username to use against Metastore database</description>
30. </property>
32. <property>
34. <name>javax.jdo.option.ConnectionPassword</name>
36. <value>123456<alue>
38. <description>password to use against Metastore database</description>
40. </property>

Download the MySQL JDBC package:
Wget http://downloads.mysql.com/archives/mysql-connector-java-5.0/mysql-connector-java-5.0.8.tar.gz.
Extract:
TAR-ZXVF mysql-connector-java-5.0.8.tar.gz

Copy the Mysql-connector-java-5.0.8-bin.jar to the hive Lib directory:
CP Mysql-connector-java-5.0.8/mysql-connector-java-5.0.8-bin.jar./lib

To start hive:
$ cd/home/zxm/hadoop/hive-0.8.1;./bin/hive

Test:
$./hive
WARNING:org.apache.hadoop.metrics.jvm.EventCounter is deprecated. Please use org.apache.hadoop.log.metrics.EventCounter the log4j.properties files.
Logging initialized using configuration in jar:file:/home/zxm/hadoop/hive-0.8.1/lib/hive-common-0.8.1.jar!/ Hive-log4j.properties
Hive History File=/tmp/work/hive_job_log_work_201207051945_218572007.txt
Hive> SHOW TABLES;
Ok
Time taken:7.281 seconds
Hive> CREATE TABLE pokes (foo INT, bar STRING);
Ok
Time taken:0.398 seconds
Hive> SHOW TABLES;
Ok
Pokes
Time taken:0.181 seconds
Hive> DESCRIBE pokes;
Ok
Foo INT
Bar string
Time taken:0.58 seconds
Hive>

Reproduced Hadoop and Hive stand-alone environment setup