Using Golang simple to achieve seven cow image processing Api_golang

has been in the use of Qiniu storage services, to generate pictures of thumbnails, blurred map, video webp, now need to move the storage to S3, then these pictures, video processing will have to write their own, this article comb the general idea.

Analyze requirements

First look at the Qiniu interface is how to deal with the picture, such as first intercept the video first second picture, and then the picture thumbnail, finally stored to a new key, commands can be so written vframe/jpg/offset/1|imageMogr2/thumbnail/400x|saveas/xxx , you can see three operations between the symbol segmentation, similar to the pipe operation of Unix.

The above operation counts as a CMD, once the API request can process multiple cmd,cmd between the semicolon split, after processing completes, returns the processing result in the callback, for example

Copy Code code as follows:

{
"id": "xxxxx",
"Pipeline": "XXX",
"Code": 0,
"desc": "The FOP was completed successfully",
"Reqid": "xtsaafnxubr5j10u",
"Inputbucket": "XXX",
"Inputkey": "xxxxx",
"Items": [
{
"cmd": "vframe/jpg/offset/1|imagemogr2/thumbnail/400x|saveas/ Zmftzs1wcml2yxrlom1vbwvudc9jb3zlci9zbmfwl3zpzgvvl2m5yzdjzjq5ltu3ngqtngzjms1izdfkltrkyjzkmzlkzwy1ni8wlza= ",
"Code": 0,
"desc": "The FOP was completed successfully",
"Hash": "Fhdn6v8ei4vw4xjgalsfxutvmeiv",
"Key": "XX",
"Returnold": 0
},
{
"cmd": "vframe/jpg/offset/1|imagemogr2/thumbnail/400x|imagemogr2/blur/45x8|saveas/ zmftzs1wcml2yxrlom1vbwvudc9jb3zlci9zbmfwl3zpzgvvl2m5yzdjzjq5ltu3ngqtngzjms1izdfkltrkyjzkmzlkzwy1ni8wlzbfymx1cg= =",
"Code": 0,
"desc": "The FOP was completed successfully",
"Hash": "Fgnirzrcsa7tzx1xvsb_4d5tiak3",
"Key": "XXX",
"Returnold": 0
}
]
}

Decomposition requirements

This program generally requires a few parts:

An HTTP interface that accepts tasks, throws them into the queue after acceptance, and returns a job ID. Worker asynchronous processing task, the number of worker and each worker parallel processing number can be configured, the worker has the retry mechanism.
From job payload to parse out the tasks that need to be done, parse out each cmd, it is best to execute each cmd in parallel, record the result of each cmd

Each cmd has multiple operation and is connected with pipe, the output of the previous operaion is the input of the latter operation

Can be divided into 1 and 2,3, 1 more independent, before writing a model of the worker, reference is this article handling 1 million Requests per Minute and go, more detailed, is used go channel as a queue , I added a beanstalk as the Providor of the queue. A further improvement is that the article only provides a set of worker numbers, and I add a parameter that sets the number of threads that each worker can execute in parallel. So here's the main answer to 3, 2.

Pipe

You can refer to this library pipe as follows:

Copy Code code as follows:

P: = pipe. Line (
Pipe. ReadFile ("Test.png"),
Resize (300, 300),
Blur (0.5),
)

Output, err: = Pipe.combinedoutput (P)
If Err!= nil {
Fmt. Printf ("%v\n", err)
}

BUF: = bytes. Newbuffer (Output)
IMG, _: = Imaging. Decode (BUF)

Imaging. Save (IMG, "Test_a.png")

or more convenient, to build a Cmd struct, using a regular match for each Operation parameter, put into a []op slice, the final implementation, struct and methods are as follows:

Copy Code code as follows:

Type CMD struct {
CMD string
SaveAs string
OPS []op
Err error
}

Type Op Interface {
Getpipe () pipe. Pipe
}

Type Resizeop struct {
width, Height int
}

Func (c Resizeop) getpipe () pipe. Pipe {
Return Resize (c.width, c.height)
}

How to use
Cmdstr: = ' file/test.png|thumbnail/x300|blur/20x8 '
CMD: = cmd{cmdstr, "Test_b.png", Nil, nil}

Cmd.parse ()
Cmd.doops ()
Sync. Waitgroup

Single cmd processing is resolved, is multiple cmd parallel problem, there is nothing good to think of, directly with sync. Waitgroup can be solved perfectly. One step at a struct, let's take a look at how this is used:

Copy Code code as follows:

Func Main () {
Cmds: = []string{}
For I: = 0; I < 10000; i++ {
Cmds = Append (Cmds, FMT. Sprintf ("cmd-%d", I))
}

Results: = Handlecmds (CMDS)

Fmt. Println (len (results))//10000
}

Func doCmd (cmd string) string {
Return FMT. Sprintf ("cmd=%s", cmd)
}

Func Handlecmds (Cmds []string) (Results []string) {
Fmt. Println (Len (CMDS))//10000
var count UInt64

    Group: = Sync. waitgroup{}
    Lock: = Sync. mutex{}
    for _, Item: = Range Cmds {
       //Count plus one
&nbs p;       Group. ADD (1)
        go func (cmd string) {
             Result: = DOCMD (cmd)
             Atomic. AddUint64 (&count, 1)

Lock. Lock ()
Results = append (results, result)
Lock. Unlock ()

Count Minus One
Group. Done ()
} (Item)
}

Blocking
Group. Wait ()

Fmt. Printf ("count=%d \ n", count)//10000
Return
}

Group essence is probably a counter, Count > 0 o'clock, Group. Wait () will block until the Count = 0. There is also a point to note that is results = append (results, result) operation is thread unsafe, clear here results is shared, need to add locks to ensure synchronization, otherwise the last Len (results) is not 10000.

We built a benchcmd to store the CMDS. As follows:

Copy Code code as follows:

Type Benchcmd struct {
Cmds []cmd
Waitgroup sync. Waitgroup
errs []error
Lock sync. Mutex
}

Func (b *benchcmd) Docmds () {
For _, Item: = Range B.cmds {
B.waitgroup.add (1)

Go func (cmd cmd) {
Cmd.parse ()
ERR: = Cmd.doops ()

B.lock.lock ()
B.errs = Append (B.errs, err)
B.lock.unlock ()

B.waitgroup.done ()
} (Item)
}

B.waitgroup.wait ()
}

The last call is like this:

Copy Code code as follows:

var Cmds []cmd
Cmd_a: = cmd{' file/test.png|thumbnail/x300|blur/20x8 ', ' test_a.png ', nil, nil}
Cmd_b: = cmd{' file/test.png|thumbnail/500x1000|blur/20x108 ', ' test_b.png ', nil, nil}
Cmd_c: = cmd{' file/test.png|thumbnail/300x300 ', ' test_c.png ', nil, nil}

Cmds = Append (Cmds, cmd_a)
Cmds = Append (Cmds, cmd_b)
Cmds = Append (Cmds, Cmd_c)

Bench: = benchcmd{
Cmds:cmds,
Waitgroup:sync. waitgroup{},
Lock:sync. mutex{},
}

Bench.docmds ()

Fmt. Println (Bench.errs)

This is just a preliminary experiment, thinking is not comprehensive enough, and just imitate api,qiniu should not do so, coupled lower, may each cmd have their own cluster of processing, that pipe this library can not solve the temporary, the current limitation is that each cmd must be in a process.