Compilation and simple deployment of C ++-implemented GFS Distributed File System kosmosfs

Like hadoop HDFS, kosmosfs is an open-source implementation of Google gfs. However, KFS is written in C ++ and currently only supports Linux and Solaris systems. Because c ++ is used for development, it must have inherent advantages over HDFS in terms of performance and stability. before studying its source code, let's take a look at how to compile and deploy it. It is very cumbersome because KFS relies on a large number of third-party software, while C ++ relies on both the operating system and the software engineering, so it requires a lot of preparation work to compile and install it. Today, I installed a KFS and wrote down the process.

First, download the source code of bytes at http://code.google.com/p/kosmosfs/downloads/list. My Linux version is centos 5.3 final and Linux kernel version is 2.6.18.

This project relies heavily on many things. We strongly recommend that you install the yum tool to reduce our workload. Which packages must be installed in advance? Below:

OpenSSL

OpenSSL-devel

Xfsprogs

Xfsprogs-devel

E2fsprogs

E2fsprogs-devel

Log4cpp

Boost

Boost-devel

Run Yum install-y [the package name above] to install the required package first. Then we start to compile kosmosfs. Suppose we place the source code in ~ /Code/KFS directory.

cd ~/code/kfsmkdir buildcd buildcmake -D CMAKE_BUILD_TYPE=RelWithDebInfo ~/code/kfs/makemake install

If the preceding steps are correct, the bin and Lib directories are generated under the build directory.

If you need to compile a client package that supports Java, you also need to install the ant tool. With yum, it is simple to execute the following directory on the top layer of Build:

yum install -y antant jar    

Check whether the build/kfs-0.5.jar and build/class directories are generated and whether they are successful.

Now we can deploy KFS. KFS is doing well. It relies on XFS to support automatic deployment. However, this automatic deployment (which only supports automatic deployment) requires you to log on to the target machine from an SSH client that can be installed on the machine without a password. Therefore, we also need to install OpenSSH:

yum install -y openssh*

When machine A needs to access machine B without a password, we need to first generate the Public Key id_rsa.pub using the ssh-keygen-t rsa command on machine B, and then copy it to the user of machine. overwrite the authorized_keys file in the SSH directory. Then we don't need to enter the SSH password, so that our installation program can go down.

To install KFS, you only need to prepare a file. Go to the KFS/scripts directory and create the machines. cfg file. The file content can be as follows:

[metaserver]node: localhostrundir: /home/leon/kfsRun/metabaseport: 20000clusterkey: test-cluster[chunkserver1]node: localhostrundir: /home/leon/kfsRun/chunk1baseport: 30000space: 10 G[chunkserver2]node: localhostrundir: /home/leon/kfsRun/chunk2baseport: 40000space: 10000 M

In this way, there is only one node, and both Chunk and meta are on this server. You must ensure that there is no password for logging on to the local machine through SSH. KFS is not so intelligent, and you will try to install the local machine through SSH.

The installation command is as follows:

scripts$ python kfssetup.py -f machines.cfg -b ../build -w ../webui -s

After successful startup, run the following command:

python kfslaunch.py -f machines.cfg --start

To view the running status, go to the KFS/build/bin/tools directory and run:

./kfsping -c -s localhost -p 30000

Check whether the service on the specified IP address and port is normal. The results are similar to the following:

Meta-server: localhost 20100Total-space: 5190.17 (MB) Used-space: 0 (MB) 

The simplest KFS has been successfully deployed. If you study the source code, you can use GDB.