Golang writing a simple image server

This is a creation in Article, where the information may have evolved or changed.

Golang writing a simple image server

Image Server

The recent development process, encountered a problem, is a large number of fragmented image storage, finally I decided to study a simple image server to solve the image file storage performance problems. Here, write a blog post documenting the process of thought and the pits I've encountered.
We know that Linux storage files are not recommended to store a large number of files in a folder, so it is not only easy to consume the inode blocks of the system, it is also easy to get the file read and write speed down quickly.

Solution Solutions

Through the analysis of requirements, you can get a solution, is to make the file randomly distributed in different folders, considering the number of file clip file 1000 is a performance threshold, you can assign the file number Fileid, by the Fileid partition, you can avoid performance problems.
To increase the overall server continuity, I decided to use UInt64 as the file ID, which would provide a larger count interval and reduce the collision probability of the new Fileid.

Basic structure

The structure is simple:
Random Fileid Generator, Fileid to file path (storage path), JSON config file read, upload, download controller

Golang using JSON to make configuration files

It's worth writing here or more,

Random number generation

The random number generation section I chose the mt19937 library written by Seehuhn, project address: github.com/seehuhn/mt19937.
It is worth mentioning that this random number library to the document is not used, a simple look at the code, found that the correct usage should be:

    mt:=mt19937.New()    mt.Seed(time.Now().UnixNano())    8)    randuint64:=mt.Uint64()

Random Fileid Generator Code

funcMakeImageID()string{    mt:=mt19937.New()    mt.Seed(time.Now().UnixNano())    var buf = make([]byte, 8)    binary.BigEndian.PutUint64(buf, mt.Uint64())    return strings.ToUpper(hex.EncodeToString(buf))}

The Fileid I generated finally came in this form: the 6a778903ad673478,16 bit hexadecimal string, which is ideal for storing in a database.

Fileid goto File path

The use of a very ugly sprintf method, do not know whether this can have a more elegant wording. It is important to note that Golang does not substring the function of intercepting substrings, but rather uses slices to intercept the substrings. The concrete way is this: Substring=string[start:end]

Func Imageid2path (imageID string) string{returnFmt. Sprintf ("%s/%s/%s/%s/%s/%s/%s/%s/%s . jpg ", Conf. storage,imageid[0:2],imageid[2:4],imageid[4:6],imageid[6:8],imageid[8:Ten],imageid[Ten: A],imageid[ A: -],imageid[ -: -])}

JSON configuration file read

Golang in the best use of JSON configuration files, people good writing, machine or parsing, the official library fully supports the reading and writing of JSON, very useful.
The JSON structure I use is as follows:

{    "ListenAddr":"0.0.0.0:10086",    "Storage":"/var/www/html/image/storage"}

Parsing code:

//corresponding JSON structure  type  Config struct  {listenaddr string  Storage string }// Generate a global conf variable store read configuration  var  conf config//read configuration function  func  loadconf () {//Open file  r, err: = OS. Open ( "Config.json" ) if  err! = nil  {log. Fatalln (Err)}//decode JSON  decoder: = JSON. Newdecoder (r) Err = decoder. Decode (&conf) if  err! = nil  {log. Fatalln (Err)}}  

After calling loadconf (), you can use the configuration in conf anywhere.

Web Interface (GORILLA/MUX)

Base Route

Golang on the most simple-to-use URL router, I think is "Github.com/gorilla/mux".
To use this requires go get, forget to say, Golang reference third-party libraries need to use go get first download library code.

Create a new Web route r: = Mux. Newrouter()//This is to set the normal get path, handlefunc the first parameter is the URL, the second parameter is to respond to the function, and finally can be in the methods string to fill in the request you want to differentiate, currently I test can be used including get, POST, put and delete R. Handlefunc("/", Homehandler). Methods("GET") r. Handlefunc("/", Uploadhandler). Methods("POST")//In the URL you can enclose a variable in curly braces, which supports placement anywhere in the URL.    It is common in restapi to write the article ID, the user ID to the URL can be implemented in this way. R. Handlefunc("/{imgid}", Downloadhandler). Methods("GET") Err: = http. Listenandserve(conf. Listenaddr, R)

How do I extract the parameters in the URL? Use in the request handler function:

    mux.Vars(r)    imageid := vars["imgid"]

R is HTTP. Request

Download processing

The code is simple and easy to see

func Downloadhandler (w http . Responsewriter, R *http . Request) {vars: = Mux. Vars (r) imageID: = Vars[ "Imgid" ] if  len  ([]rune (imageID))! = 16  {w.write ([]byte  (" Error:imageid incorrect. "))    return } Imgpath: = Imageid2path (imageID) if ! FileExist (Imgpath) {w.write ([]byte  ( "Error:image not F        Ound. "))    return } http . Servefile (W, R, Imgpath)}  

Golang HTTP package, for the server to provide file download implemented a very good encapsulation, an HTTP. Servefile (W, R, FilePath) is done, where W and R are the parameters passed in the processing function, we only need to pass in as is, FilePath is required to provide to the client file, the rest of the HTTP library will be automatically processed, including Content-type and the like.

Upload processing

Func Uploadhandler (whttp. Responsewriter, R *http. Request) {//randomly generate a non-existent Fileidvar imgidstring     for{Imgid=makeimageid ()if! FileExist (Imageid2path (Imgid)) {Break}}//upload parameter is UploadFileR.parsemultipartform ( +<< -)file, _, Err: = R.formfile ("UploadFile")ifErr! = Nil {Log. PRINTLN (Err) w.write ([]byte("Error:upload Error."))return} deferfile. Close ()//detection file typeBuff: = make ([]byte, +) _, Err =file. Read (Buff)ifErr! = Nil {Log. PRINTLN (Err) w.write ([]byte("Error:upload Error."))return} filetype: =http. Detectcontenttype (Buff)iffiletype!="Image/jpeg"{W.write ([]byte("Error:not JPEG."))return}//Wrap file pointer    Log. Println (filetype)if_, Err =file. Seek (0,0); err!=nil{Log. PRINTLN (ERR)}//Create an entire storage tree in advance (if you do not create a storage tree, the following file creation will not succeed)    ifErr=buildtree (Imgid); err!=nil{Log. PRINTLN (ERR)}//Writes the file to the imageID specified locationF, err: = OS. OpenFile (Imageid2path (imgid), OS. O_wronly|os. O_create,0666)ifErr! = Nil {Log. PRINTLN (Err) w.write ([]byte("Error:save Error."))return} defer f.close () io. Copy (F,file) W.write ([]byte(Imgid))}

Summarize

Basic development is like this, very simple we use Golang to achieve a high-performance Web image server, Golang use its unique mechanism to achieve the implementation of high concurrency without too much special processing to complete the development of the program. If you use Golang correctly, you can easily do the high-performance parallel network programming that the common language is very cumbersome to handle.
Ps:golang http built-in high concurrency support, if you write a program to use go for high concurrency, you can use the Golang go keyword, in front of your function with a go flag, he will automatically go to the coprocessor run, very convenient, Regardless of how you create the thread, the Golang kernel automatically helps you do it.
Code
Code uploaded github:https://github.com/zjyl1994/quickimageserver
There's a chance I'll increase the ability to generate thumbnails in the cloud.