Fuse-DFS installation Configuration

Introduction

Fuse-DFS is a function module of the hadoop project. Map Data on DFS to a specified local mount point. Because existing software cannot directly use data on HDFS, you can use fuse to map local file systems.

 

Fuse-DFSDependent libraries include

1: fuse. Ko

Generally, this kernel module is provided in Linux. If not, compile this module. Use sudo lsmod | grep fuse to determine whether fuse is automatically loaded. You can use the insmod or modprobe command to load this module.

 

2: FUSE (http://fuse.sourceforge.net /)

Fuse library needs to be compiled and installed by yourself (for detailed compilation methods, refer to fuse-2.8.4/readme)

$./Configure

$ Make

$ Sudo make install

The compiled fuse library is installed in the/usr/local/lib directory by default. Modify Environment VariablesLD_LIBRARY_PATH, Add the directory/usr/local/lib.

 

3: libhdfs

The module of the hadoop project. Compilation Method:

$ Ant compile-dcompile. c ++ = true-dlibhdfs = true

 

Note: hadoop compilation processThe project downloads the dependent libraries through the network. If the system cannot be connected to the Internet, you can modify hadoopBuild. XML in the project directoryFile, there are multiple places to be modified,Modify the attribute ivy_repo_urlIs

<Property name = "ivy_repo_url" value = "file :// $ {User. Home}/install/ivy-2.0.0-rc2.jar "/>

Local repository ivy-2.0.0-rc2.jarFileThe path is actually modified. In addition, other dependent libraries can also directly download the downloaded. ivy2Folder to homeDirectory.

 

4: fusedfs

$ Ln-S $ hadoop_home/build/C ++/Linux-amd64-64/Lib $ hadoop_home/build/libhdfs

$ Export LD_LIBRARY_PATH =

/Usr/lib:/usr/local/lib: $ hadoop_home/build/C ++/Linux-amd64-64/lib: $ java_home/JRE/lib/amd64/Server

$ Ant compile-contrib-dlibhdfs = 1-dfusedfs = 1

 

Note:Linux-amd64-64Yes 64Bit SystemLibhdfsDirectory generated after compilation, amd64Yes 64Bit system JavaDependent library Directory, which must be modified according to the actual situation. LD_LIBRARY_PATHChanges to environment variables should be recorded in the logon environment variables.

 

Start DFS-FuseProcess

The startup command is

CD $ hadoop_home/build/contrib/fuse-DFS

./Fuse_dfs_wrapper.sh DFS: // master-Host: 9000 $ home/fuse-DFS

Replace master-host with the Host Name of the master node corresponding to hadoop.

Make sure that the fuse-DFS folder is empty in the $ home directory.