Tachyon 0.7.1 Pseudo-distributed cluster installation and testing

Tachyon is a highly fault-tolerant Distributed file system that allows files to be reliably shared in the cluster framework at the speed of memory, just like Spark and MapReduce. By leveraging information inheritance, memory intrusion, Tachyon gains high performance. The Tachyon working set file is cached in memory and allows different jobs/queries and frameworks to access the cache file at the speed of memory. As a result, tachyon can reduce the number of data sets that need to be used frequently by accessing the disk.

SOURCE download

Source Address: Https://github.com/amplab/tachyon
git clone https://github.com/amplab/tachyon.git

Compile

Enter Tachyon_home

Git tagv0.1.0v0.2.0v0.2.1v0.3.0v0.4.0v0.4.1v0.4.1-thriftv0.5.0v0.6.0v0.6.1v0.6.2v0.6.3v0.6.4v0.7.0v0.7.0-rc1v0.7.112345678910 111213141516171819

switch to v0.7.1 version

git checkout v0.7.1 note:checking out ' v0.7.1 '. You're in the ' detached HEAD ' state. You can look around, make experimentalchanges and commit them, and you can discard any commits do in thisstate Witho UT impacting any branches by performing another checkout.  If you want to create a new branch to retain commits your create, you maydo so (now or later) by Using-b with the checkout Command again. Example:git Checkout-b New_branch_namehead is currently located in A2b69c9 ... [Maven-release-plugin] Prepare release v0.7.11234567891011121314

Execute the following command:

Export maven_opts= "-xmx2g-xx:maxpermsize=512m-xx:reservedcodecachesize=512m" mvn clean package-dhadoop.version= 2.3.0-cdh5.1.0-dspark.version=1.4.1-dskiptests12

Configuration

1. Create a journal Directory
   Mkdir-p/home/cluster/apps/tachyon/journal

2. Create RAMDisk
   Mkdir-p/home/cluster/apps/tachyon/ramdisk

3. Modify Tachyon_home/conf/tachyon-env.sh.template to Tachyon_home/conf/tachyon-env.sh
   MV Conf/tachyon-env.sh.template conf/tachyon-env.sh

4. Edit tachyon_home/conf/tachyon-env.sh Add the following content

Export Java_home=/home/cluster/share/java1.7export Hadoop_home=/home/cluster/apps/hadoopexport TACHYON_HOME=/home /cluster/apps/tachyonexport tachyon_ram_folder= $TACHYON _home/ramdiskexport tachyon_master_address=master12345

1. To modify the Tachyon log storage path:
   Create logs directory under tachyon_home/
   mkdir logs
   Vim Conf/log4j.properties adds the following content:
   Tachyon.logs.dir=/home/cluster/apps/tachyon/logs

2. Modify Tachyon_home/conf/core-site.xml.template to Tachyon_home/conf/core-site.xml
   MV Conf/core-site.xml.template Conf/core-site.xml
   To edit the Conf/core-site.xml, add the following:
   
<configuration> 
<property> 
　 <name>fs.tachyon.impl</name> 
　　<value>tachyon.hadoop.TFS</value> 
　</property> 
</configuration> 


3. Formatting Tachyon

Bin/tachyon formatconnecting to localhost as Wangyue ... Formatting Tachyon Worker @ wangyue-umconnection to localhost closed. Formatting Tachyon Master @ master12345

Start

bin/tachyon-start.sh localkilled 0 processes on wangyue-umkilled 0 processes on wangyue-umconnecting to localhost as wangy Ue... Killed 0 processes on wangyue-umconnection to localhost closed. [sudo] password for wangyue:formatting Ramfs:/home/cluster/apps/tachyon/ramdisk (1GB) Starting Master @ masterstarting W Orker @ wangyue-umjps: 21025 ZeppelinServer22214 Worker21987 Master21360 TachyonWorker32212 RemoteInterpreterServer23361 NameNode29603 RemoteInterpreterServer21324 TachyonMaster23493 DataNode21412 Jps27005 Remoteinterpreterserver has started Tachyonmaster & TachyonWorker12345678910111213141516171819202122232425

The tachyon-start.sh Local command initiates both the master and worker processes locally. It is important to note that running the TACHYON-START.SH local command must have the password to switch root, and no one will be able to start. This is because RAMFS requires root permissions for formatting.

Test

webui:http://master:19999

/home/cluster/apps/tachyon$ bin/tachyon runtest basic cache_through/default_tests_files/ basicfile_cache_through has been removed2015-09-09 20:00:17,911 info     (MasterClient.java:connect)  - Tachyon client  (version 0.7.1)  is  trying to connect with master @ master/127.0.0.1:199982015-09-09  20:00:17,938 info    (MasterClient.java:connect)  - user registered with  the master @ master/127.0.0.1:19998; got UserId 52015-09-09  20:00:17,964 info    (CommonUtils.java:printTimeTakenMs)  - createfile with  fileId 3 took 57 ms.2015-09-09 20:00:17,999 INFO    ( WorkerClient.java:connect)  - Trying to get local worker host :  Wangyue-um.local2015-09-09 20:00:18,013 info    (WorkerClient.java:connect)  - Connecting local worker @  wangyue-um.local/10.32.24.78:299982015-09-09 20:00:18,098 info    (BlockOutStream.java : Get)  - writing with local stream. tachyonfile: /default_tests_files/ basicfile_cache_through, blockindex: 0, optype: cache_through2015-09-09 20:00:18,136  INFO    (CommonUtils.java:createBlockPath)  - folder /home/cluster/apps/ tachyon/ramdisk/tachyonworker/5 was created!2015-09-09 20:00:18,140 info    (localblockoutstream.java:<init>)  - /home/cluster/apps/tachyon/ramdisk/tachyonworker/5/ 3221225472 was created! tachyonfile: /default_tests_files/basicfile_cache_through,  blockindex: 0, blockid: 3221225472, blockcapacitybyte: 5368709122015-09-09  20:00:18,203 info   (CommonUtils.java:printTimeTakenMs)  - writefile to file /default_tests_files/ basicfile_cache_through took 238 ms.2015-09-09 20:00:18,246 info    ( CommonUtils.java:printTimeTakenMs)  - readfile file /default_tests_files/basicfile_cache_ Through took 43 ms. passed the test! perform   bin/tachyon runTests  cycle test 123456789101112131415

Stop it

[Email protected]:/home/cluster/apps/tachyon$ bin/tachyon-stop.sh killed 1 processes on wangyue-umkilled 1 processes on W Angyue-umconnecting to localhost as Wangyue ... Killed 0 processes on wangyue-umconnection to localhost closed. Perform jps:21025 ZeppelinServer22214 Worker21987 Master32212 Re moteInterpreterServer23361 NameNode29603 RemoteInterpreterServer27469 Jps23493 DataNode27005 Remoteinterpreterserver no Tachyon related processes 123456789101112131415161718

Respecting originality, refusing to reprint, http://blog.csdn.net/stark_summer/article/details/48321605

Tachyon 0.7.1 Pseudo-distributed cluster installation and testing