Fuse Mount hadoop2.0.0 File System hdfs to local

I. Environment configuration 1. host system: RedHatEnterpriseLinux6.1 (X86_64) Hadoop version: hadoop-2.0.0-cdh4.3.0JDK version: jdk1.6.0 _ 45Ant version: apache-ant-1.8.2Maven version: apache-maven-3.1.02. environment requirements: automake, autoconf, m4, libtool, pkgconfig, f

I. Environment configuration 1. host system: RedHat Enterprise Linux 6.1 (X86_64) Hadoop version: hadoop-2.0.0-cdh4.3.0 JDK version: jdk1.6.0 _ 45 Ant version: apache-ant-1.8.2 Maven version: apache-maven-3.1.0 2. environment requirements: automake, autoconf, m4, libtool, pkgconfig, f

I. Environment Configuration

1. host system: RedHat Enterprise Linux 6.1 (X86_64)

Hadoop version: hadoop-2.0.0-cdh4.3.0

JDK version: jdk1.6.0 _ 45

Ant version: apache-ant-1.8.2

Maven: apache-maven-3.1.0

2. environment requirements:

Automake, autoconf, m4, libtool, pkgconfig, fuse, fuse-devel, fuse-libs

3. Download the compilation tool and set environment variables:

(You need to download ant 1.8.2 and maven 3.1.0 externally. Download them from the apache website)

Installation Process:

# Wget http://mirror.bjtu.edu.cn/apache//ant/binaries/apache-ant-1.8.2-bin.zip
# Export JAVA_HOME =/usr/java/jdk1.6.0 _ 45
# Export HADOOP_HOME =/usr/hadoop/hadoop-2.0.0-cdh4.3.0.
# Export OS _ARCH = amd64
### I386/amd64 (optional)
# Export OS _BIT = 64
### 32/64 (optional)
# Export ANT_HOME =/root/apache-ant-1.8.2

# Export PATH = $ PATH: $ ANT_HOME/bin

# Export PATH =/root/apache-maven-3.1.0/bin: $ PATH
Note: different versions of hadoop may cause many problems, mainly because there are many file directories under it, and the paths for storing files are different. If your version is different from mine, you need to find the file path by yourself

# Yum-y install fuse-devel fuse-libs

# Modprobe fuse (Mount: modprobe view: lsmod uninstall: rmmod)

#### The preceding content can also be added to the environment variable file/etc/profile ####

Ii. Compile libhdfs

1. You can test whether ant and mvn have been configured before compiling libhdfs:

# Ant-version

The prompt is as follows:
Apache Ant (TM) version 1.8.2 compiled on December 20 2010

# Mvn-version

The prompt is as follows:

Apache Maven 3.1.0 (893ca28a1da9d5f51ac03827af98bb730128f9f2;

10:15:32 + 0800)
Maven home:/root/apache-maven-3.1.0
Java version: 1.6.0 _ 45, vendor: Sun Microsystems Inc.
Java home:/usr/java/jdk1.6.0 _ 45/jre
Default locale: en_US, platform encoding: UTF-8
OS name: "linux", version: "2.6.32-131.0.15.el6.x86 _ 64", arch: "amd64", family: "unix"


2. Go to the installation path:/usr/hadoop/hadoop-2.0.0-cdh4.3.0/src/hadoop-mapreduce1-project

Compile libhdfs. Because libhdfs requires cross-compilation, an error is reported when compiling directly into src. Therefore, ant compilation is required.

Note: OS _ARCH and OS _BIT must be set; otherwise, they will fail.

Ant compile-Dlibhdfs = 1-Dcompile. c ++ = 1


Problems:

(1) The ivy.2.2.0.jar package cannot be downloaded because the mvnrepo url set in the build. xml file has an error:

Solution:

(2). If the problem occurs during compilation (network problem), you can ant clean and re-compile it. The problem in the hadoop2.0.0. configuration file is:

Solution:

The url of reactor. repo needs to be defined:

Add the following in the/usr/hadoop/hadoop-2.0.0-cdh4.3.0/src/hadoop-mapreduce1-project/ivy/ivysettings. xml file:

Value = "http://repo1.maven.org/maven2"
Override = "false"/>

3. After compilation is successful, the prompt is as follows:

........................................

Compile:

BUILD SUCCESSFUL
Total time: 9 minutes 38 seconds

........................................

Create a dynamic Connection Library:

Find the libhdfs. so directory. Someone on the libhdfs. so website said it was generated after compiling libhdfs, but after compiling, only two static libraries (libhadooppipes. a and libhadooputils. a) were generated ). So it comes directly with hadoop, my version of hadoop-2.0.0-cdh4.3.0 is in the $ HADOOP_HOME/lib/native directory (if not, you can download it online: https://github.com/cloudera/impala/tree/master/thirdparty/hadoop-2.0.0-cdh4.3.0 ), then copy all the library files under its directory to $ HADOOP_HOME/src/hadoop-mapreduce1-project/build/c ++/Linux-amd64-64/lib and then create the soft connection as follows

Note: Follow the hadoop version to find the corresponding directory and create a link:

Ln-s/usr/hadoop/hadoop-2.0.0-cdh4.3.0/src/hadoop-mapreduce1-project/build/c ++/Linux-amd64-64/lib

/Usr/hadoop/hadoop-2.0.0-cdh4.3.0/src/hadoop-mapreduce1-project/build/libhdfs

Add the path of the compiled dynamic library file (file ending with. so) to the dynamic connection:

# Export LD_LIBRARY_PATH = $ JAVA_HOME/jre/lib/amd64: $ HADOOP_HOME/lib/:/usr/local/lib

:/Usr/lib:/usr/hadoop/hadoop-2.0.0-cdh4.3.0/src/hadoop-mapreduce1-project/build/c ++/Linux-amd64-64/lib

3. Compile fuse-dfs:

(1) Compile:

Ant compile-Dlibhdfs = 1-Dfusedfs = 1

If the compilation fails, the possible cause is that libhdfs cannot be found. Check the libhdfs path carefully. See the ln-s in step 1, after compilation is successful, the fuse_dfs executable program is generated in the fuse-dfs/src directory.

(2) test:

A. Configure the environment, edit/etc/fuse. conf, and write the following content
User_allow_other
Mount_max = 1, 100

B. Edit $ HADOOP_HOME/src/hadoop-mapreduce1-project/build/contrib/fuse-dfs/fuse_dfs_wrapper.sh

If ["$ HADOOP_HOME" = ""]; then
Export HADOOP_HOME =/usr/hadoop/hadoop-2.0.0-cdh4.3.0/src/hadoop-mapreduce1-project/
Fi

# ****************** 2013/9/13 ******************** **
Export PATH = $ HADOOP_HOME/src/hadoop-mapreduce1-project/src/contrib/fuse-dfs/src: $ PATH
#*************************************** **********
If ["$ OS _ARCH" = ""]; then
Export OS _ARCH = amd64
Fi

If ["$ JAVA_HOME" = ""]; then
Export JAVA_HOME =/usr/java/jdk1.6.0 _ 45
Fi

If ["$ LD_LIBRARY_PATH" = ""]; then
Export LD_LIBRARY_PATH = $ JAVA_HOME/jre/lib/amd64: $ HADOOP_HOME/src/hadoop-mapreduce1-project/build/libhdfs/: $ HADOOP_HOME/share/hadoop/hdfs /: /usr/local/lib:/usr/lib
Fi

# ************** If dev build set paths accordingly *****************
If [-d $ HADOOP_HOME/src/hadoop-mapreduce1-project/build]; then
For f in $ {HADOOP_HOME}/src/hadoop-mapreduce1-project/build/*. jar; do
Export CLASSPATH = $ CLASSPATH: $ f
Done
For f in $ HADOOP_HOME/src/hadoop-mapreduce1-project/build/ivy/lib/Hadoop/common/*. jar; do
Export CLASSPATH = $ CLASSPATH: $ f
Done
Export PATH =$ HADOOP_HOME/src/hadoop-mapreduce1-project/src/contrib/fuse-dfs: $ PATH
Fi

./Fuse_dfs $ @

(3) Mount hdfs with fuse:

$ Mkdir/tmp/dfs
$ Cd $ HADOOP_HOME/build/contrib/fuse-dfs
$./Fuse_dfs_wrapper.sh dfs: // hadoopmaster: 9000/tmp/dfs
Note: port = 9000, server = hadoopmaster

Success prompt: INFO fuse_options.c: 162 Adding FUSE arg/tmp/dfs

Note: If you have created a directory on hadoop or uploaded a file, the corresponding directory file will be displayed.

(4) uninstall hdfs:

Fusermount-u/tmp/dfs or umount/tmp/dfs

Problems:

(1) An error is reported when hdfs is mounted:
./Fuse_dfs: error while loading shared libraries: libhdfs. so.0.0.0: cannot open shared object file: No such file or directory

Solution:
A. cannot find the dynamic library: ldd fuse_dfs
B. Add the absolute path of the unfound Library to the configuration file of the ld query:/etc/ld. so. conf
C. Run ldconfig to recreate/etc/ld. so. cache

Iv. Test commands

? Hdfsclient write
Dd if =/dev/zero bs = 4096 count = % d | hadoop fs-put-% s/% s-% d. dat
? Hdfsclient read
Hadoop fs-get % s/f % 03d. dat->/dev/null
? Fuse write
Dd if =/dev/zero bs = 4096 count = % d of = % s/% s-% d. dat
? Fuse read
Dd if = % s/f % 03d. dat bs = 4096 of =/dev/null