Open source Cloud Computing Technology Series (v) (Rise of the Black Horse Sector/sphere actual combat article)

In the heyday of Java Hadoop, open source cloud computing has a black-sector/sphere based on C + +, which challenges Hadoop in terms of performance, open Cloud Consortium (OCC) opened the Cloud Computing Association Cloud testbed Open Cloud experimental bed software test, sector is about twice as fast as Hadoop.

This article first on this black horse to do a practical exercise, first feel, the next in-depth design principle, explore the nature of cloud computing.

Oct is a set of computing clusters for multiple data centers spanning a core 10G bandwidth education Network.

Implemented in 2 phases:

Phase 1. Phase 1 is operational in June 2008 and consists of the ' a ', ' cores distributed, across four cities in the U.S. This is upgraded in September and 2008 to cores.

This is diagram of the testbed. The Phase 1 equipment consists of four racks. Each rack contains nodes. Each node has 4 cores. The racks are located in:

University of Illinois at Chicago (Chicago) Starlight (Chicago) Calit2 (La Jolla) Johns Hopkins University (Baltimore )

All of the racks are connected by a wide area GB/s network.

Phase 2. Phase 2 of the "Open Cloud testbed is planned to" operational by June, 2009. The testbed would add 4 racks of equipment for a total of 8 racks containing over 1000 cores. In addition, two more sites'll is connected by the GB/s networks. Phase 2 Racks is located at:

Johns Hopkins University (Baltimore) Calit2 (La Jolla) MIT Lincoln Lab (Cambridge, Pittsburgh supercompter) Negie Mellon University (Pittsburgh) Starlight (Chicago) University of Illinois at Chicago (Chicago)

In addition, in Phase 2, the Open Cloud testbed'll add shared, non-dedicated resources.

Enterprise and University Union, open source cloud computing domain scale expands increasingly. The aforementioned Hadoop is also one of the software used by Oct, where we focus on another dark horse sector/sphere, which is also one of the core software used in Oct, it can be seen that its components, sector/sphere focus on the operation with the public network, emphasizing the core data security, In addition to skilled C + + developers to provide open source cloud computing technology framework. Let's take a look at the Black Horse, which is twice times more than Hadoop in the performance test.

Sector/sphere design idea is clear, but the data and documents are relatively few, this also to large-scale promotion brings inconvenient

Before experiencing this black horse, let's take a look at the design structure of the sector/sphere.

There is a security server setting when it comes to a more prominent place in sector, which provides a guarantee of cloud computing on a wide area network.

The software is very small, download the latest version of codeblue.1.23c.tar.gz, some issues can be discussed in the forum.

http://sourceforge.net/forum/?group_id=172838

Before you install make, check that several basic packages inside the Debian OS are installed.

Libssl-dev,gcc,g++,libfuse-dev If you are ready to experience fuse functionality.

debian:~# Tar xvzf codeblue.1.23c.tar.gz

debian:~/codeblue2/conf# ls
client.conf master_node.cert masters.list security_node.key slave.conf topology.conf
master.conf master_node.key security_node.cert slave_acl.conf slaves.list users
debian:~/codeblue2/conf# pwd
/root/codeblue2/conf

Change the security,master,slave,client configuration file according to the environment you are deploying.

The configuration file is very clear, basically change the corresponding host, and the data directory can be.

debian:~/codeblue2/conf# more master.conf
#SECTOR Server port number
Sector_port
6000

#security server Address
Security_server
localhost:5000

debian:~/codeblue2/conf# more slave.conf
#Master Address
Master_address
localhost:6000

#Data Directory
Data_directory
/root/data/

debian:~/codeblue2/conf# more client.conf
#Master Address
Master_address
localhost:6000

Compile, make successful completion, you can start the service.

To start the service:

debian:~/codeblue2/security#/sserver &
[1] 8637
debian:~/codeblue2/security# Sector security server running at Port 5000

The server is started successfully; There is no further to output from this program. Please don't shutdown the security server; Otherwise no client may is able to login. If the server is down for any reason, can restart it without restarting the Masters

debian:~/codeblue2/security# CD ... /master/
debian:~/codeblue2/master#/start_master &
[2] 8638
debian:~/codeblue2/master# Sector Master is successfully running now. Check Sector.log for more details.
There is no further screens output from this program.

debian:~/codeblue2/master# CD ... /slave/
debian:~/codeblue2/slave# ls
Copying serv_file.cpp serv_spe.cpp slave.cpp slave.o start_slave.cpp
Makefile serv_file.o SERV_SPE.O slave.h start_slave
debian:~/codeblue2/slave#/start_slave &
[3] 8652
debian:~/codeblue2/slave# scaning/root/data/
This sector slave are successfully initialized and running now.
Slave Process:gmp 47087 DATA 42064

debian:~/codeblue2/slave#

The default sector will retain 10GB of space, the resulting test data is also 10GB, if you want to use a small amount of data to verify, you can change the source code to achieve.

For example, if you need to generate 100M of test data to sort.

So

VI randwriter.cpp

Modify and remove the last 00, which reduces the amount of test data from 10GB to 100M.

10GB = 100 * 1000000
(long int i = 0; i < 1000000 + i)
58 {
Keygen (record);
Ofs.write (record, 100);
61}

A for (long int i = 0; i < 1000001 + i)
68 {
A long long int d = i * 100;
Idx.write ((char*) &d, 8);
71}

and mrsort.cpp inside need to comment out a paragraph, otherwise run not past.

debian:~/codeblue2/client/examples# VI mrsort.cpp

/* IF (3!= argc)
{
cout << "Usage:mrsort" << Endl;
return 0;
}
*/

Then make or make Clean,make in the Codeblue2 directory.

So the following test can start, also will not explode your hard drive, but play cloud computing, suggest that you or more reserved some hard disk, many benchmark procedures to the default amount of data to achieve a certain level can be representative, but also to reflect the huge Izumo, hehe.

Generate test data.

debian:~/codeblue2/client/examples#./testfs
recv cmd 127.0.0.1 6000 type
recv cmd 127.0.0.1 6000 type 103
recv cmd 127.0.0.1 6000 Type a
===> start file server 127.0.0.1 6000
Open file Tmp/guide.dat 127.0.0.1 60 833
Rendezvous Connect source 127.0.0.1 45180/root/data//tmp/guide.dat
Connected
file server closed 127.0.0. 1 45180 0
127.0.0.1 6000 14,/tmp/guide.dat,0,1245914942,4
recv cmd 127.0.0.1 6000 type
===> star T file Server 127.0.0.1 6000
Rendezvous Connect source 127.0.0.1 45180/root/data//tmp/guide.dat.idx
Connected
open file tmp/guide.dat.idx 127.0.0.1 60833
File server closed 127.0.0.1 45180 0
127.0.0.1 6000 ide.dat.idx,0,1245914943,16
Start time 1245914943
JOB 4 1
1 spes found! 1 data seg total.
recv cmd 127.0.0.1 6000 type 203
Starting SPE ... 0 45180 Randwriter 3
Rendezvous connect 127.0.0.1 45180
Connected
Connect SPE 127.0.0.1 3
New Job/tmp/guide.dat 0 1
completed 127.0.0.1 46922
sending data back ... 0
127.0.0.1 6000 21,test/sort_input.0.dat,0,1245914946,100000000
127.0.0.1 6000 25,test/sort_ input.0.dat.idx,0,1245914946,8000008
recv cmd 127.0.0.1 6000 type
Comp Server closed 127.0.0.1 46922 2
Rep Ortsphere 127.0.0.1 6000 3

View sector system information through./sysinfo.

debian:~/codeblue2/client/tools#/sysinfo
Sector System Information:
Running since Thu June 25 03:28:39 2009
Available Disk Size 27413 MB
Total File Size 102 MB
Total number of Files 2
Total number of Slave Nodes 1
------------------------------------------------------------
Total number of clusters 4
cluster_id total_nodes availdisk (MB) FileSize (MB) netin (MB) netout (MB)
0:1 27413 102 0 0
1:0 0 0 0 0
2:0 0 0 0 0
3:0 0 0 0 0
------------------------------------------------------------
slave_id IP TS (US) availdisk (MB) totalfile (MB) Mem (MB) CPU (US) netin (MB) netout (MB)
1:127.0.0.1 1245915399257411 27413 102 0 3440000 0 0

debian:~/codeblue2/client/tools#/ls/
Test <dir>
debian:~/codeblue2/client/tools#/ls/test
Sort_input.0.dat 100000000 bytes Thu June 25 03:29:06 2009
Sort_input.0.dat.idx 8000008 bytes Thu June 25 03:29:06 2009

You can see that the test data has been generated.

Use TESTDC to do sequencing experiments.

debian:~/codeblue2/client/examples#/TESTDC
Start Time 1245915520
JOB 100000000 1000000
Request Shuffler 127.0.0.1 41406
1 Spes found! 1 Data seg total.
Connect SPE 127.0.0.1 5
Stage 1 accomplished 1245915552
JOB 100000000 1000000
2 Spes found! Data seg total.
Connect SPE 127.0.0.1 6
Connect SPE 127.0.0.1 7
Stage 2 accomplished 1245915557
SPE COMPLETED
debian:~/codeblue2/client/examples#

In running a wordcount example, this has a corresponding example example in Hadoop.

debian:~/codeblue2/client/tools#./mkdir html
debian:~/codeblue2/client/tools#/upload Mv.cpp
Usage:upload <src file/dir> <dst dir>
debian:~/codeblue2/client/tools#/upload mv.cpp/html
Uploading Mv.cpp of 1821 bytes
Open File/html/mv.cpp 127.0.0.1 60833
Uploading accomplished! AVG Speed 0.0121632 MB/s.

debian:~/codeblue2/client/tools# CD ... /examples/
debian:~/codeblue2/client/examples#/wordcount
Start Time 1245915644
JOB 1821-1
Request Shuffler 127.0.0.1 41406
1 Spes found! 1 Data seg total.
Connect SPE 127.0.0.1 10
Stage 1 accomplished 1245915645
SPE COMPLETED
debian:~/codeblue2/client/examples#

Interested students can visit http://sector.sourceforge.net/to get more information.