Distributed memory file System Alluxio Combat

Objective

  Span style= "Font-family:arial, Helvetica, Sans-serif;" >      Alluxio is a distributed memory file system that accesses the files in the Alluxio in a cluster with the ability to access memory. The Alluxio is architected at the bottom of the Distributed file storage and the upper Tachyon 。

Alluxio originated in the PhD project at UC Berkeley Amplab Laboratory during the Li Haoyuan of Alluxio company founder. Since the release of the first open source version of Alluxio, the project has grown rapidly. The number of community contributors has rapidly increased to more than 200, more than 200 people from more than 50 companies, including international giants such as Barclays,ibm,intel and so on. In addition, more companies have deployed Alluxio to their own production environment.

Official website: http://www.alluxio.org/

650) this.width=650; "src="/e/u261/themes/default/images/spacer.gif "style=" Background:url ("/e/u261/lang/zh-cn/ Images/localimage.png ") no-repeat center;border:1px solid #ddd;" alt= "Spacer.gif"/>650 "this.width=650;" src= "http ://s4.51cto.com/wyfs02/m01/8b/59/wkiom1hkjwzbetiwaafr3woiiaw600.png "title=" 1.png "alt=" Wkiom1hkjwzbetiwaafr3woiiaw600.png "/>

Planning

Master 10.64.8.3

Slave 10.64.8.3, 10.64.8.4, 10.64.8.5

One: Installation

• Slave1, Slave2, Slave3

(1) Installing the JDK

$ wget http://download.oracle.com/otn-pub/java/jdk/8u112-b15/jdk-8u112-linux-x64.tar.gz$ Tar XF Jdk-8u112-linux-x64.tar.gz-c/opt$ vim/etc/profile Add export java_home=/opt/jdk1.8.0_112export PATH= $PATH: $JAVA _home /binexport classpath= $JAVA _home/lib/dt.jar: $JAVA _home/lib/tools.jar$ source/etc/profile

(2) Installation Alluxio

$ cd opt$ wget http://downloads.alluxio.org/downloads/files/1.3.0/alluxio-1.3.0-bin.tar.gz$ Tar XF alluxio-1.3.0-bin.tar.gz$ CD alluxio-1.3.0

II: Cluster configuration

• Slave1, Slave2, Slave3

1: Use Conf/alluxio-env.sh.template to copy a configuration.

$ CP conf/alluxio-env.sh.template conf/alluxio-env.sh#!/usr/bin/env Bash#alluxio installation directory alluxio_home=/opt/ alluxio-1.3.0# Log directory (default: ${alluxio_home}/logs). Host name of the #ALLUXIO_LOGS_DIR =/var/log/alluxio# master Alluxio_master_ hostname=slave1# MASTER node IP, which will be discarded in the V2 version of the configuration # alluxio_master_address=10.64.8.3# worker node to save the directory of In-memory data (Default:/mnt/ RAMDisk). #ALLUXIO_RAM_FOLDER # The address of the underlying storage system, which defaults to the local file system (default: ${alluxio_home}/underfsstorage) #ALLUXIO_UNDERFS_ address# the memory settings of the worker node. (DEFAULT:1GB) alluxio_worker_memory_size= 50gb# under JMX remote Debugging # alluxio_java_opts# alluxio_master_java_opts# ALLUXIO_WORKER_JAVA_OPTS # alluxio_user_java_opts

2: Configure worker Node

$ vim conf/workers10.64.8.310.64.8.410.64.8.5

Three: Start up

(1) Format Alluxio log and Work program storage directory.

• Slave1, Slave2, Slave3

$./bin/alluxio format

(2) Start the master node and use the local parameter to start both master and worker.

• Slave1

$./bin/alluxio-start.sh Local

650) this.width=650; "src="/e/u261/themes/default/images/spacer.gif "style=" Background:url ("/e/u261/lang/zh-cn/ Images/localimage.png ") no-repeat center;border:1px solid #ddd;" alt= "Spacer.gif"/>650 "this.width=650;" src= "http ://s5.51cto.com/wyfs02/m02/8b/56/wkiol1hkknct9cjcaaa5gf3os68356.png "title=" 2.png "alt=" Wkiol1hkknct9cjcaaa5gf3os68356.png "/>

(3) Start worker node

• Slave2, Slave3

$./bin/alluxio-start.sh Worker

650) this.width=650; "src=" Http://s3.51cto.com/wyfs02/M00/8B/56/wKioL1hKKQLiNKygAAAjy5G2QG4383.png "title=" 3.png " alt= "Wkiol1hkkqlinkygaaajy5g2qg4383.png"/>

650) this.width=650; "src="/e/u261/themes/default/images/spacer.gif "style=" Background:url ("/e/u261/lang/zh-cn/ Images/localimage.png ") no-repeat center;border:1px solid #ddd;" alt= "Spacer.gif"/>

Four: Access

(1) visited http://localhost:19999 see Alluxio's Master

650) this.width=650; "src=" Http://s2.51cto.com/wyfs02/M00/8B/59/wKiom1hKKRuiflZcAAF9pJpFJBA463.png "title=" 4.png " Width= "height=" 312 "border=" 0 "hspace=" 0 "vspace=" 0 "style=" width:700px;height:312px; "alt=" Wkiom1hkkruiflzcaaf9pjpfjba463.png "/>

(2) You can see all the files in the Alluxio.

650) this.width=650; "src=" Http://s1.51cto.com/wyfs02/M00/8B/56/wKioL1hKKU-ysJuoAAF90pU2LvU112.png "title=" 5.png " Width= "height=" 207 "border=" 0 "hspace=" 0 "vspace=" 0 "style=" WIDTH:700PX;HEIGHT:207PX; "alt=" Wkiol1hkku-ysjuoaaf90pu2lvu112.png "/>

(3) The worker can see the status of all the workers

650) this.width=650; "src=" Http://s3.51cto.com/wyfs02/M00/8B/56/wKioL1hKKXfClPSkAAExan2E5sM561.png "height=" 347 " Style= "border:1px solid rgb (221,221,221); width:700px;height:347px;background:no-repeat 50% 50%;" Width= "alt=" Spacer.gif "title=" 6.png "border=" 0 "hspace=" 0 "vspace=" 0 "/>

(4) Visit http://localhost:30000 to see information about the current worker.

650) this.width=650; "src=" Http://s3.51cto.com/wyfs02/M01/8B/56/wKioL1hKKb7ByTVyAAFayfRjjZI504.png "height=" 376 " Style= "border:1px solid rgb (221,221,221); width:700px;height:376px;background:no-repeat 50% 50%;" Width= "alt=" Spacer.gif "title=" 7.png "border=" 0 "hspace=" 0 "vspace=" 0 "/>

V: Use the Alluxio command line

(1) View Aullio, use local address port by default

$./bin/alluxio fs ls alluxio://10.64.8.3:19999/$./bin/alluxio FS LS/

650) this.width=650; "src=" Http://s5.51cto.com/wyfs02/M01/8B/59/wKiom1hKKeHARHkuAAAsRuzU08k063.png "title=" 8.png " alt= "Wkiom1hkkeharhkuaaasruzu08k063.png"/>

(2) from the local file system copy file to Alluxio

$./bin/alluxio FS copyfromlocal/opt/cassandra/conf//cassandracopied/opt/cassandra To/cassandra

650) this.width=650; "src=" Http://s2.51cto.com/wyfs02/M02/8B/59/wKiom1hKKe_S8ZCZAAAlMDmfWRs792.png "title=" 9.png " alt= "Wkiom1hkke_s8zczaaalmdmfwrs792.png"/>

(3) Cat command can output file contents

$./bin/alluxio FS Cat/test.txt

650) this.width=650; "src=" Http://s3.51cto.com/wyfs02/M01/8B/56/wKioL1hKKf_SiPFkAAAffsnzhag757.png "title=" 10.png "alt=" Wkiol1hkkf_sipfkaaaffsnzhag757.png "/>

(4) Copy files from Alluxio to local

$./bin/alluxio FS Copytolocal/602.txt/tmp/602.txtcopied/602.txt To/tmp/602.txt

650) this.width=650; "src=" Http://s1.51cto.com/wyfs02/M02/8B/56/wKioL1hKKhGD_cTTAAAwILRWkCQ518.png "title=" 11.png "alt=" Wkiol1hkkhgd_cttaaawilrwkcq518.png "/>

(5) Persisting Alluxio data to local

Alluxio data is in memory, by default using a local filesystem as the underlying system, you can persist the data in the Alluxio to the local file system. The path is specified by alluxio_underfs_address and the default is./underfsstorage/

$./bin/alluxio fs persist/cassandra$ ls./underfsstorage/cassandra

(6) Common supported commands are the same as for Linux system command usage.

./bin/alluxio FS Cat./bin/alluxio FS Chmod./bin/alluxio FS Chown./bin/alluxio FS Cp./bin/alluxio FS Mv./bin/alluxio FS RM. /bin/alluxio FS Touch./bin/alluxio FS mkdir

VI: Use NFS as the underlying file system

(1) Mount NFS on Master

$/etc/init.d/rpcbind start$ mount-t nfs-o vers=3,tcp 10.10.10.10:/vol-01/alluxio/mnt/nfs

(2) Modify ALLUXIO configuration

$ vim./conf/alluxio-env.sh#alluxio_underfs_address= converted to Alluxio_underfs_address=/mnt/nfs

(3) Formatting

$./bin/alluxio format

(4) Start

Master

$./bin/alluxio-start.sh Master

Worker

$./bin/alluxio-start.sh Worker

(5) test

Create a file in NFS

$ echo 11111111111 >>/mnt/nfs/nfstest.txt

See Alluxio to see this file.

$./bin/alluxio fs ls/-rw-r--r--root root 12.00B 11-08-2016 17:22:34:477 not in Memory/nfstest.txt$./bin/allux IO fs cat/nfstest.txt11111111111

Seven: using HDFs as the underlying file system

(1) Alluxio version

The default alluxio-1.3.0-bin.tar.gz installation package supports HDFs 2.2.0, and with other versions of HDFs you need to download the corresponding Alluxio installation package. or recompile with the default package as follows:

1: Modify Pom.xml, use Hadoop2.2

2: Recompile
$ mvn Clean package-dskiptests
A new Alluxio-assemblies-1.3.0-jar-with-dependencies.jar is generated in the Assembly/target directory after the compilation is successful
(2) Alluxio configuration
1: Modify conf/alluxio-env.sh
$ vim conf/alluxio-env.shalluxio_underfs_address=hdfs://10.64.8.1:8020
(3) Formatting
$./bin/alluxio format
(4) Start
Master
$./bin/alluxio-start.sh Master
Worker
$./bin/alluxio-start.sh Worker
(5) View
$./bin/alluxio LS/
found that the files in HDFs are in Alluxio.
650) this.width=650; "src=" Http://s5.51cto.com/wyfs02/M00/8B/56/wKioL1hKKl7ieyPcAABdyjXBV-4257.png "title=" 12.png "alt=" Wkiol1hkkl7ieypcaabdyjxbv-4257.png "/>

650) this.width=650; "src=" Http://s2.51cto.com/wyfs02/M01/8B/56/wKioL1hKKn6DSrnQAAGvypCSLl0544.png "title=" 13.png "Width=" "height=" 269 "border=" 0 "hspace=" 0 "vspace=" 0 "style=" WIDTH:750PX;HEIGHT:269PX; "alt=" Wkiol1hkkn6dsrnqaagvypcsll0544.png "/>

Eight: mount Alluxio with Fuse

 
 

  
  
 
   
Linux kernel 2.6.9 and above
 
   
JDK 1.8 and above
 
   
Libfuse 2.9.3 and above (2.8.3 can also work, but will prompt for some warnings)
 
   
Need to recompile according to the source package, need JDK, MAVEN, Libfuse
 
  
 
 

(1) Configuring the JDK
$ wget http://download.oracle.com/otn-pub/java/jdk/8u112-b15/jdk-8u112-linux-x64.tar.gz$ Tar XF Jdk-8u112-linux-x64.tar.gz-c/opt$ vim/etc/profile Add export java_home=/opt/jdk1.8.0_112export PATH= $PATH: $JAVA _home /binexport classpath= $JAVA _home/lib/dt.jar: $JAVA _home/lib/tools.jar$ source/etc/profile

(2) Configuring MAVEN
$ wget http://mirrors.cnnic.cn/apache/maven/maven-3/3.0.5/binaries/apache-maven-3.0.5-bin.tar.gz$ TAR-ZXVF Apache-maven-3.0.5-bin.tar.gz$vim/etc/profile Add export maven_home=/opt/apache-maven-3.0.5export PATH= $PATH: $MAVEN _home/bin$ Source/etc/profile

(2) Installation Libfuse
$ wget http://mirror.centos.org/centos/6/os/x86_64/packages/fuse-libs-2.8.3-5.el6.x86_64.rpm$ RPM-IVH fuse-libs-2.8.3-5.el6.x86_64.rpm

(3) Compile and install
$ git clone git://github.com/alluxio/alluxio.git$ cd alluxio$ mvn install-dskiptests---Need download file if it appears Java.lang.Out Ofmemoryerror:java heap Space, please execute: $ export maven_opts= "-xmx2g-xx:maxpermsize=512m-xx:reservedcodecachesize=512m"

(4) Using Alluxio-fuse
In the case of HDFs as the underlying storage system, so is the other storage.
Formatting
$./bin/alluxio format
Start
$./bin/alluxio-start.sh Worker
Mount
$ mkdir-p/data/test$./bin/alluxio-fuse.sh mount/data/test
650) this.width=650; "src="/e/u261/themes/default/images/spacer.gif "style=" Background:url ("/e/u261/lang/zh-cn/ Images/localimage.png ") no-repeat center;border:1px solid #ddd;" alt= "Spacer.gif"/>650 "this.width=650;" src= "http ://s2.51cto.com/wyfs02/m02/8b/56/wkiol1hkkq6i6emvaablmjok3gs131.png "title=" 14.png "alt=" Wkiol1hkkq6i6emvaablmjok3gs131.png "/>

This article is from the "Sauce paste" blog, please make sure to keep this source http://heqin.blog.51cto.com/8931355/1881114
Distributed memory file System Alluxio Combat