Tachyon basically uses 08 ----- running hadoop mapreduce on tachyon

1. Modify the hadoop configuration file

1. Modify the core-site.xml File

Add the following attributes so that mapreduce jobs can use the tachyon file system as input and output.

<property> <name>fs.tachyon.impl</name> <value>tachyon.hadoop.TFS</value></property>

2. Configure hadoop-env.sh

Add environment variables for the tachyon client jar package path at the beginning of the hadoop-env.sh file.

export HADOOP_CLASSPATH=/usr/local/tachyon/client/target/tachyon-client-0.5.0-jar-with-dependencies.jar

3. Synchronize the modified configuration file to other nodes.

[[email protected] hadoop]# scp hadoop-env.shnode2:/usr/local/hadoop/etc/hadoop/hadoop-env.sh                                                                                                                        100%3499     3.4KB/s   00:00   [[email protected] hadoop]# scp hadoop-env.shnode3:/usr/local/hadoop/etc/hadoop/hadoop-env.sh                                                                                                                        100%3499     3.4KB/s   00:00   [[email protected] hadoop]# scp core-site.xmlnode2:/usr/local/hadoop/etc/hadoop/core-site.xml                                                                                                                        100% 1421     1.4KB/s   00:00   [[email protected] hadoop]# scp core-site.xmlnode3:/usr/local/hadoop/etc/hadoop/core-site.xml                                                                                                                        100% 1421     1.4KB/s   00:00   [[email protected] hadoop]#

4. Start all zookeeper nodes

[[email protected] hadoop]# zkServer.sh startJMX enabled by defaultUsing config:/usr/local/zookeeper/bin/../conf/zoo.cfgStarting zookeeper ... STARTED[[email protected] hadoop]# ssh node2 zkServer.shstartJMX enabled by defaultUsing config:/usr/local/zookeeper/bin/../conf/zoo.cfgStarting zookeeper ... STARTED[[email protected] hadoop]# ssh node3 zkServer.shstartJMX enabled by defaultUsing config:/usr/local/zookeeper/bin/../conf/zoo.cfgStarting zookeeper ... STARTED[[email protected] hadoop]#

5. Start the hadoop Cluster

[[email protected] hadoop]# pwd/usr/local/hadoop[[email protected] hadoop]# sbin/start-all.sh

6. Start the tachyon Cluster

[[email protected] hadoop]# tachyon-start.sh all Mount

7. Start tachyon master on node2

[[email protected] ~]# tachyon-start.sh masterStarting master @ node2[[email protected] ~]#

8. View Processes

[[email protected] conf]# jps21954 QuorumPeerMain22398 JournalNode24120 TachyonWorker22765 NodeManager22572 DFSZKFailoverController22663 ResourceManager24009 TachyonMaster24354 Jps22216 DataNode22115 NameNode[[email protected] conf]# ssh node2 jps15524 NameNode16538 TachyonWorker15880 NodeManager15802 DFSZKFailoverController16650 Jps15592 DataNode15456 QuorumPeerMain15683 JournalNode16598 TachyonMaster[[email protected] conf]# ssh node3 jps9294 DataNode9231 QuorumPeerMain9382 JournalNode10050 Jps10007 TachyonWorker9476 NodeManager[[email protected] conf]#

Ii. Test mapreduce jobs

1. Upload the test file to tachyon

[[email protected] conf]# tachyon tfs copyFromLocal /etc/passwd /passwdCopied /etc/passwd to /passwd[[email protected] conf]# tachyon tfs tail /passwdtp:/sbin/nologinnobody:x:99:99:Nobody:/:/sbin/nologin

2. Execute mapreduce jobs

[[email protected] hadoop]# pwd/usr/local/hadoop[[email protected] hadoop]# hadoop jarshare/hadoop/mapreduce/hadoop-mapreduce-examples-2.2.0.jar wordcount -libjars/usr/local/tachyon/client/target/tachyon-client-0.5.0-jar-with-dependencies.jartachyon://node1:19998/passwd tachyon://node1:19998/out

3. view the output result after execution is complete.

[[email protected] hadoop]# tachyon tfs cat/out/part-r-00000Daemon:/var/cache/rpcbind:/sbin/nologin     1Daemon:/var/run/pulse:/sbin/nologin    1IPv4LL       1NFS  1SSH:/var/empty/sshd:/sbin/nologin         1Service     1Stack:/var/lib/avahi-autoipd:/sbin/nologin      1System     1User:/var/ftp:/sbin/nologin     1User:/var/lib/nfs:/sbin/nologin        2

This article is from the "tachyon" blog, please be sure to keep this source http://ucloud.blog.51cto.com/3869454/1564201

Tachyon basically uses 08 ----- running hadoop mapreduce on tachyon