Installation and application of Golang Open source Distributed File System Weed-fs

This is a creation in Article, where the information may have evolved or changed.

1. Installation Configuration Golang Development environment

Go is a compiled, parallel, and garbage-collected programming language developed by Google, available to http://www.golangtc.com/ Download to download the go1.7rc6.linux-amd64.tar.gz, download and unzip the package to the specified directory (example:/usr/local/).

Tar-zxvf/usr/local/src/go1.7rc6.linux-amd64.tar.gz-c/usr/local/

Configure the environment variables, edit the/etc/profile file, and add the following:

Gohome=/usr/local
Export goroot= $GOHOME/go
Export gopath=/usr/webserver/go/
Export path= $PATH: $GOROOT/bin
Export path= $PATH: $GOPATH/bin

Save exit, execute source/etc/profile Reload environment variable, execute go version, if you can display the version information of go, the configuration is successful.

2. Download and install Weed-fs

1 Way One: Download the source code compilation installation (to VPN some packages will not be, I was directly downloaded compiled) Weed-fs is also known as SEAWEED-FS, is a simple and high-performance distributed storage System, he has the following features:
A. Storing massive files
B. Quick access to stored files
Refer to Weed-fs's official website Https://github.com/chrislusf/seaweedfs Installation Guide installation Weed-fs, see:

After executing the go get github.com/chrislusf/seaweedfs/go/weed command, the WEED-FS is downloaded and compiled, and the generated executable weed is placed in the $gopath/bin directory, the source is stored in the $GOPATH/SRC directory, Because some Web sites in the domestic wall, may appear some packages can not download errors, directly on-line download to the corresponding directory is good;

The following error:
root@ubuntu:/tmp/chengxf# Go get github.com/chrislusf/seaweedfs/go/weed
Package Github.com/chrislusf/seaweedfs/go/weed
Imports Bazil.org/fuse
Imports Bazil.org/fuse/fs
Imports golang.org/x/net/context:unrecognized Import Path "Golang.org/x/net/context"
# CD.; Git Clone Https://code.google.com/p/go-decimal-inf.exp/tmp/chengxf/gopath/src/speter.NET/go/exp/math/dec/inf
Cloning into '/tmp/chengxf/gopath/src/speter.net/go/exp/math/dec/inf ' ...
error:failed to connect to 2404:6800:400a:805::200e:network are unreachable while accessing Https://code.google.com/p/go -decimal-inf.exp/info/refs
Fatal:http request failed
Package Github.com/chrislusf/seaweedfs/go/weed
Imports Bazil.org/fuse
Imports Bazil.org/fuse/fs
Imports Github.com/boltdb/bolt
Imports Github.com/chrislusf/raft
Imports Github.com/gogo/protobuf/proto
Imports Github.com/dgrijalva/jwt-go
Imports GITHUB.COM/GOCQL/GOCQL
Imports GITHUB.COM/GOLANG/GROUPCACHE/LRU
Imports Github.com/golang/snappy/snappy
Imports Speter.net/go/exp/math/dec/inf:exit Status 128
Package Github.com/chrislusf/seaweedfs/go/weed
Imports Bazil.org/fuse
Imports Bazil.org/fuse/fs
Imports Github.com/boltdb/bolt
Imports Github.com/chrislusf/raft
Imports Github.com/gogo/protobuf/proto
Imports Github.com/dgrijalva/jwt-go
Imports GITHUB.COM/GOCQL/GOCQL
Imports GITHUB.COM/GOLANG/GROUPCACHE/LRU
Imports Github.com/golang/snappy/snappy
Imports Github.com/golang/protobuf/proto
Imports Github.com/syndtr/goleveldb/leveldb
Imports Github.com/syndtr/goleveldb/leveldb/util
Imports Github.com/syndtr/gosnappy/snappy
Imports Gopkg.in/redis.v2
Imports Gopkg.in/bufio.v1
Imports Github.com/disintegration/imaging
Imports golang.org/x/image/bmp:unrecognized Import Path "Golang.org/x/image/bmp"
Package Github.com/chrislusf/seaweedfs/go/weed
Imports Bazil.org/fuse
Imports Bazil.org/fuse/fs
Imports Github.com/boltdb/bolt
Imports Github.com/chrislusf/raft
Imports Github.com/gogo/protobuf/proto
Imports Github.com/dgrijalva/jwt-go
Imports GITHUB.COM/GOCQL/GOCQL
Imports GITHUB.COM/GOLANG/GROUPCACHE/LRU
Imports Github.com/golang/snappy/snappy
Imports Github.com/golang/protobuf/proto
Imports Github.com/syndtr/goleveldb/leveldb
Imports Github.com/syndtr/goleveldb/leveldb/util
Imports Github.com/syndtr/gosnappy/snappy
Imports Gopkg.in/redis.v2
Imports Gopkg.in/bufio.v1
Imports Github.com/disintegration/imaging
Imports golang.org/x/image/tiff:unrecognized Import Path "Golang.org/x/image/tiff"
root@ubuntu:/tmp/chengxf#


First, the above unrecognized import path "Imports Golang.org/x/xxx/xxx", you can manually go https://github.com/golang download the corresponding package, placed and/TMP/CHENGXF The corresponding subdirectory of the/GOPATH/SRC, after executing again, finds that only the following error is left:
root@ubuntu:/tmp/chengxf# Go get github.com/chrislusf/seaweedfs/go/weed
# CD.; git clone https://code.google.com/p/go-decimal-inf.exp/tmp/chengxf/gopath/src/speter.net/go/exp/math/dec/inf
Cloning into '/tmp/chengxf/gopath/src/speter.net/go/exp/math/dec/inf ' ...
error:failed to connect to 2404:6800:4005:80b::200e:network are unreachable while accessing Https://code.google.com/p/go -decimal-inf.exp/info/refs
Fatal:http request failed
Package Speter.net/go/exp/math/dec/inf:exit Status 128
root@ubuntu:/tmp/chengxf#


Above is to get code.google.com on the resources on the error, you can manually go to https://github.com/go-inf/inf/tree/master download the source package, and extract to/tmp/chengxf/gopath/src/ Speter.net/go/exp/math/dec/inf directory, perform go get github.com/chrislusf/seaweedfs/go/weed again error:
root@ubuntu:/tmp/chengxf/gopath/src/speter.net/go/exp/math/dec/inf# Go get github.com/chrislusf/seaweedfs/go/weed
.. /.. /.. /.. /.. /.. /github.com/gocql/gocql/helpers.go:13:2: Code in Directory/tmp/chengxf/gopath/src/speter.net/go/exp/math/dec/inf expects import "Gopkg.in/inf.v0"


Well, I don't know why, anyway, I'll/tmp/chengxf/gopath/src/speter.net/go/exp/math/dec/inf/dec.go the file "//import" Gopkg.in/inf.v0 "" Delete it and then recompile it with Go get github.com/chrislusf/seaweedfs/go/weed. The executable file weed is generated under the/tmp/chengxf/gopath/bin directory.

2. Mode two: Download the compiled to run the installation

Download Address:https://github.com/chrislusf/seaweedfs/releases/tag/0.74

Can directly download its running package, directly decompression, you can run (I do so, if so, etc/profile inside the Gopath this item will not need)

Export gopath=/usr/webserver/go/
Export path= $PATH: $GOPATH/bin

Linux under command download:

Wgethttps://bintray.com/artifact/download/chrislusf/seaweedfs/weed_0.70beta_linux_amd64.tar.gz

Tar zxvf weed_0.70beta_linux_amd64.tar.gz

Service startup:

Service startup:


1 into the weed of the decompression directory, mine is this.


root@ubuntu:~/webserver/weed_0.70beta_linux_amd64#./weed Master



Note that it is started with a service, the port is 9333, and this is its default port.


2 Starting the File storage service

root@ubuntu:~/webserver/weed_0.70beta_linux_amd64#./weed volume-dir= "/root/webserver/weed_0.70beta_linux_amd64/ Data "-max=5-mserver=" localhost:9333 "-port=9080 &

3 Commit A storage request, this time weed first to assign a global file ID

Curl-x POST Http://localhost:9333/dir/assign





4 Storing a picture


Curl-x put-f file=@/root/webserver/weed_0.70beta_linux_amd64/error.jpg http://127.0.0.1:9080/5,019d90e98a





5 Access Test

Because my server address is 192.168.1.47,http://192.168.1.47:9080/5,019d90e98a.jpg.









Even, you can add dimensions, dynamically get the size of the corresponding picture






Really good. If you look at the above, you will find its file ID, which is always there. This is the unique identification of the file. In fact, this ID is every 3 parts exist.


where "FID": "5,019d90e98a" is fid,fid composed of three parts "Volumeid, Needleid, Cookie".


Volumeid:1 the ID of a physical volume that is stored in 32bit


NEEDLEID:01 64bit globally unique Needleid, each stored file is different (except for each backup).


cookie:9d90e98a 32bit cookie value, for security purposes, to prevent malicious attacks.


All of this can only be used as a Java developer in the way of curl. Or you need to encapsulate one of these APIs for easy access.

The above is the general start: We can also configure the parameters at startup

Cases:
Prepare: Create the following folder under the/TMP/CHENGXF directory: Mdata, vol1, vol2, input, and output, and create a new test file a.txt and B.xml in the input folder.
Start Master:weed master-mdir= "/tmp/chengxf/mdata"-defaultreplication= "001"-ip= "10.124.69.235"-port=9334 &
Start volume:weed volume-dir=/tmp/chengxf/vol1/-mserver= "10.124.69.235:9334"-ip= "117.78.1.100"-ip.bind= " 10.124.69.235 "-port=8081-max=5 &
Weed volume-dir=/tmp/chengxf/vol2/-mserver= "10.124.69.235:9334"-ip= "117.78.1.100"-ip.bind= "10.124.69.235"-port= 8082-max=5 &
Storage file: Weed upload-server= "117.78.1.100:9334"-dir= "/tmp/chengxf/input"-include=*.txt
Return file after execution fid:[{"fileName": "/tmp/chengxf/input/a.txt", "FileUrl": "117.78.1.100:8081/1,01eb8f1e42", "FID": " 1,01eb8f1e42 "," Size ": 44}]
Access files according to FID: Weed download-server= "117.78.1.100:9334"-dir= "/tmp/chengxf/output" 1,01eb8f1e42
A.txt file found in/tmp/chengxf/output directory after execution
117.78.1.100 is a public network IP, through the firewall mapped to the intranet machine 10.124.69.235, can not directly bind the public address, so the use of Ip.bind bound intranet address to achieve.

Reference article: Http://blog.csdn.net/qq286210662/article/details/46328153?ref=myread

Reference article: http://sanwen.net/a/tabgsbo.html