Web Development notes using the Go language framework

This is a creation in Article, where the information may have evolved or changed.

It is recently necessary to crawl its users ' images with the Instagram API, which needs to be validated by Oauth2, so the app must contain a web interface. Imagine being able to return the number of downloads in real time, so use WebSocket. There is a need to consider the efficiency of the problem, a combination of the above, want to use a language development, the final choice to use Golang development, node callback really do not like.

Objective

There are many frameworks for web development of Golang, such as Martini, Gin, Revel,gorilla, and so on. Previously played revel , felt too much encapsulation, as a small application does not need to be so complicated, and Google to get the result is revel efficiency is relatively poor. ginThe benchmark display is 40 times times more efficient than Martini, but the gin is relatively new, so his ecological circle is less. The final choice, there are a martini lot of middleware can choose, including the WebSocket, and behind the use of Gorilla WebSocket this package.

Interface and Features

1. A link to the Oauth2 Landing authorization page

2. After the authorization is completed, jump back to the service page, at this time get the Access_token, you can do whatever you like. All the functions are also focused on this page, the final interface as shown.

点击连接is used to open the WebSocket connection. 开始发送数据is to start sending the user ID to the server, and the server invokes the API to start fetching the image. 停止used to stop the crawl service for this time. 已完成数量used to return the number of crawled images in real time.

Approximate structure of the program

Here Jobs ,,, and so on the whole goroutine #1 #2 is to be in the websocket after the disconnection, the download can continue to execute. websocket goroutineis the scope after the connection is established, the Goroutine does not exist after the connection has been disconnected. Jobs, which NextUrl acts as a queue role. Doneis just a count. There are less than two global variables written here, and the Quit chan int IsPreparing bool two variables are used to let the front-end control whether the crawler is working.

A simple understanding is a for loop that generates a task, a for loop for a consuming task, and a for loop to return a count to the client. It has to be lamented that the design of the Goroutine channel makes the coding simple and straightforward.

Problems encountered

Because the first serious use of go, still encountered a lot of problems. But the demand is relatively simple, so there is no contact with what in-depth content. The main focus is on the problem posed by the strong type.

DB Query

Before writing an article about Database/sql, this time directly using sqlx This library, you can write a lot less code, but also less mistakes. But after all, not as laravel so convenient, fortunately need to write a few SQL, temporary write a few methods will be done. At the same time think about how to implement a eloquent API. Seems to be difficult.

JSON processing

The strong type determines the processing of JSON is a pain. Previously wrote a weather forecast applet, using map[string]*json.RawMessage this mapping structure, and then one layer to solve the JSON. It was not known at the time that this was problematic, because if used RawMessage , the quotation marks of the strings are " also preserved, so that the string result is preceded by quotation marks.
Once again Google once again, found still need to use map[string]interface{} to map, and then use type assertion a layer of to solve the JSON. This is a painful process, think of PHP in json_decode() tears.

Stop Goroutine

How to interrupt a goroutine is a problem because you need control to start stopping. Google will soon have results.

    go func() {        for {            select {            case <-Quit:                IsPreparingJobs = false                return            default:                // to do something            }        }    }()

Here set one IsPreparingJobs is used for interrupts after starting this loop again.

Testing

The testing tools provided by Golang are very convenient and go test can be tested. Two common methods are copied from Martini source.

func expect(t *testing.T, a interface{}, b interface{}) {    if a != b {        t.Errorf("Expected %v (type %v) - Got %v (type %v)", b, reflect.TypeOf(b), a, reflect.TypeOf(a))    }}func refute(t *testing.T, a interface{}, b interface{}) {    if a == b {        t.Errorf("Did not expect %v (type %v) - Got %v (type %v)", b, reflect.TypeOf(b), a, reflect.TypeOf(a))    }}

Summarize

Feel Golang as a web development tool, in the data format processing, there is no weak type language convenient. This node is very good, json to object is very convenient. Perhaps with Promise,node will be more useful. Golang also has advantages, goroutine is very useful, the official library function is very full, packaged as a binary executable file makes deployment exceptionally easy, strong type language efficiency is high.

In the last few days of the Shadowsockets incident, as a SS user, in addition to thank the selfless developers, the rest is just anger and disappointment. Yesterday, I saw Lao Luo's T1 conference,, in adhering to the principle of life Born to be proud 天生骄傲 , Lao Luo has been my role model. Look forward to the Hammer conference tonight.