Distributed File System (KFS) source code compilation and Installation Process

Kosmos Distributed File System (KFS) is a storage system specially designed for data-intensive applications (such as search engines and data mining), similar to Google's GFS and hadoop's HDFS distributed file system. KFS is implemented using C ++. Supported clients include C ++, Java, and python.

Previously in the open-source KFS on the basis of the development, here to introduce how to KFS source code compilation and installation process (take kfs-0.5 as an example ).

1. Install dependent Software

To compile and run the KFS file system, install the following software packages:

• Boost (1.34 or above)
• Cmake (2.4.6 or later)
• Log4cpp (1.0 or above)
• GCC version (4.1 or more)
• XFS devel RPMs on Linux

The following describes the process of manually installing the software package. Here we assume that you have downloaded the compressed file of the software package.

1. Install gcc

 
1 tar zxvf gcc-4.1.2.tar.gz
-Mkdir/usr/local/gcc-4.1.2
3 CD gcc-4.1.2
4. /configure -- prefix =/usr/local/The gcc-4.1.2 -- enable-threads = POSIX -- disable-checking -- enable -- long -- Host = i386-redhat-linux -- With-system-zlib -- enable- ages = C, c ++, Java
5 make
6 make install
 

Note: The above process will install GCC in the/usr/local/gcc-4.1.2 directory, support C/C ++ and Java language, other options refer to the GCC help instructions.

2. Install log4cpp

 
1 tar xvzf log4cpp-1.0.tar.gz
2 CD log4cpp-1.0
3./configure
4 make
5 make install
 

3. Install XFS

 
1 tar xvzf xfsprogs-3.0.1.tar.gz
2 CD xfsprogs-3.0.1
3./configure
4 make
5 make install
6 make install-Dev
 

4. Install cmake

 
1 tar xvzf cmake-2.6.4.tar.gz
2 CD cmake-2.6.4
3./Bootstrap
4 make
5 make install
 

5. Install boost

 
1 tar xvzf boost_1_39_0.tar.gz
2 CD boost_000039_0
3./Bootstrap. Sh
4./bjam-stools = GCC -- without-Python install
 

2. Compile and install KFS

First, download kfs-0.5.tar.gz (click to download) to sourceforge. assume that the downloaded directory is the user root directory.

Decompress the KFS package. Assume that the KFS directory is :~ /Kfs-0.5, compile the source code Directory :~ /Kfs-0.5/build.

To fully compile KFS, the following three parts are required:

• Compile the C ++ part to generate metaserver/chunkserver, tools for various API operations, and C ++ client;
• Compile the Java part and generate KFS. jar, that is, the Java client, to call the local C ++ method through Java JNI;
• Compile the python extension module to generate the python client.

2.1 compile C ++

 
1 tar xvzf kfs-0.5.tar.gz
2 CD kfs-0.5
3 mkdir build
4 CD build
5 cmake-D cmake_build_type = relwithdebinfo ~ Kfs-0.5
6 gmake
7 gmake install
 

After compilation, the binary file will be installed in the following directory:

• Executable File :~ Kfs-0.5/build/bin
  
• Library File :~ /Kfs-0.5/build/lib
  

Note: Debug debugging information is included in the KFS compilation process.

2.2 compile Java

 
1 CD ~ Kfs-0.5
2 ant jar
 

After compilation, the generated file is:

• Java class file :~ /Kfs-0.5/build/classes
• Java jar package :~ /Kfs-0.5/build/kfs-0.5.jar

Finally, add the generated jar package to the classpath environment variable:

 
1 Export classpath =$ {classpath }:~ /Kfs-0.5/build/kfs-0.5.jar
 

2.3 compile Python extension module

To compile the python extension module, Step 2.1 is required to generate the KFS client library file. The directory of the library file is :~ /Kfs-0.5 /.

To compile the python extension module, follow these steps:

 
1 CD ~ /Kfs-0.5/src/CC/Access
2 edit kfs_setup.py and setup the include path. Specifically, kfsext = extension ('kfs', include_dirs ['kfs/src/CC/',' <path to boost> '])
3 Python kfs_setup.py ~ /Kfs-0.5/build/lib/build
 

After compilation, generate the shared link library KFS. So for installation:

 
1 Python kfs_setup.py ~ /Kfs-0.5/build/lib/install
 

If you want to install it in a specified directory (for example ~ /Kfs-0.5/build/LIB), you need to set the home option:

 
1 Python kfs_setup.py ~ /Kfs-0.5/build/lib install -- home = ~ /Kfs-0.5/build/lib
 

In addition, you must update the environment variables pythonpath and LD_LIBRARY_PATH:

 
1 Export pythonpath =$ {pythonpath }:~ /Kfs-0.5/build/lib/lib64/Python
2 Export LD_LIBRARY_PATH =$ {LD_LIBRARY_PATH }:~ /Kfs-0.5/build/lib
 

3. Start the KFS service process

1. metaserver

1 CD ~ /Kfs-0.5/build/src/CC/meta
2 CP ~ /Kfs-0.5/CONF/metaserver. GP ./
3./metaserver. GP
 

Note: Create the logdir and cpdir Directories specified in metaserver. GP in advance:./kfslog and./kfscp.

2. chunkserver

 
1 CD ~ /Kfs-0.5/build/src/CC/chunk
2 CP ~ /Kfs-0.5/CONF/chunkserver. GP ./
3./chunkserver.
 

3. kfsclient

 
1 CD ~ /Kfs-0.5/build/src/CC/tools
2./kfsshell-S <meta Server Name>-P <port>
3./cptokfs-S <meta Server Name>-P <port>-D <Source Path>-k <KFS path>
4./cpfromkfs-S <meta Server Name>-P <port>-D <Source Path>-k <KFS path>
 

Note :~ The/kfs-0.5/build/src/CC/tools directory has a variety of API operation tools, such as kfsshell, cptokfs, cpfromkfs, etc., here is not a list.

4. View and disable KFS service processes

1. View Processes

1 PS-Aux | grep metaserver
2 PS-Aux | grep chunkserver
3 Ps-Aux | grep kfsshell
 

2. Close the process

 
1 kill-9 <pid>
 

5. References

• Http://code.google.com/p/kosmosfs/

• Http://sourceforge.net/projects/kosmosfs/