After one months of study Golang

This is a creation in Article, where the information may have evolved or changed.

Problems encountered

Connection pool. Because PHP does not have a connection pool, when high concurrency there will be a large number of database connections directly impact on MySQL, eventually causing the database to hang out. Although Swoole has a connection pool, but Swoole is just an extension of PHP, before using the Swoole process has stepped a lot of pits. After our discussion, we still feel that using Golang is more controllable.

Selection of Frames

YAF is always used in PHP, so it's natural to choose gin in go. Because we have always been the principle: as close as possible to the underlying code.

An overly well-packaged framework is not conducive to mastering and understanding the entire system. I don't need you to tell me what this directory is for, how to write this configuration, how to use this function, and so on.

Gin is a lightweight routing framework that fits our needs. In order to develop better, we have also made several middleware.

Middleware--input

Each interface needs to get a GET or post parameter, but the gin comes with a method that only returns a string, so we have a simple encapsulation. After encapsulation we can switch directly to the desired data type as needed.

package inputimport (    "strconv")type I struct {    body string}func (input *I) get(p string) *I {    d, e := Context.GetQuery(p)    input.body = d    if e == false {        return input    }    return input}func (input *I) post(p string) *I {    d, e := Context.GetPostForm(p)    input.body = d    if e == false {        return input    }    return input}func (input *I) String() string {    return input.body}func (input *I) Atoi() int {    body, _ := strconv.Atoi(input.body)    return body}

package input//获取GET参数func Get(p string) *I {    i := new(I)    return i.get(p)}//获取POST参数func Post(p string) *I {    i := new(I)    return i.get(p)}

Before encapsulation

pid, _ := strconv.Atoi(c.Query("product_id"))alias := c.Query("product_alias")

After encapsulation

  pid := input.Get("product_id").Atoi()  alias := input.Get("product_alias").String()

Middleware--logger

Gin own logger is relatively simple, generally we need to write the log by date sub-file to a directory. So we rewrote a logger ourselves, and this logger can be implemented to divide the log by date and send the error message to Sentry.

  Package Ginximport ("FMT" "IO" "OS" "Time" "Github.com/gin-gonic/gin" "Sao.cn/configs") Var ( LogPath string Lastday int) func init () {LogPath = configs. Load (). Get ("Sys_log_path"). (string) _, Err: = OS. Stat (LogPath) if err! = nil {os. Mkdir (LogPath, 0755)}}func defaultwriter () io. Writer {Writercheck () return gin. Defaultwriter}func defaulterrorwriter () io. Writer {Writercheck () return gin. Defaulterrorwriter}func Writercheck () {nowday: = time. Now (). Day () if nowday! = lastday {var file *os. File FileName: = time. Now (). Format ("2006-01-02") LogFile: = Fmt. Sprintf ("%s/%s-%s.log", LogPath, "Gosapi", filename) file, _ = os. Create (logFile) if file! = Nil {gin. Defaultwriter = File gin. defaulterrorwriter = file}} lastday = Nowday}  

Package Ginximport ("bytes" "Encoding/json" "Errors" "FMT" "io" "Net/url" "Time" github.com/gin- Gonic/gin "" Gosapi/application/library/output "" Sao.cn/sentry ") func Logger () gin. Handlerfunc {return Loggerwithwriter (Defaultwriter ())}func loggerwithwriter (Outwrite io. Writer) gin. Handlerfunc {return func (c *gin. Context) {Newlog (c). Captureoutput (). Write (Outwrite). Report ()}}const (level_info = "INFO" Level_warn = "warning" level_error = "ERROR" level_fatal = "FATAL" ) type Log struct {startAt time. Time ConText *gin.       Context writer Responsewriter Error Error level string time string ClientIP string Uri String Paramget URL. Values ' JSON: "Pget" ' parampost URL. Values ' JSON: "Ppost" ' Respbody string timeuse string}func newlog (c *gin. Context) *log {bw: = responsewriter{buffer:bytes. Newbufferstring (""), responsewriter:c.writer} c.writer = &AMP;BW ClientIP: = c.clientIP () Path: = C.request.url.path Method: = C.request.method Pget: = C.request.url.query () var ppost URL. Values if method = = "POST" {c.request.parseform () Ppost = c.request.postform} return &log{sta Rtat:time. Now (), Context:c, WRITER:BW, Time:time. Now (). Format (time. RFC850), Clientip:clientip, Uri:path, Paramget:pget, Parampost:ppost}}func (L *log) captureoutput () *Log {L.conTex T.next () O: = new (output. O) JSON. Unmarshal (L.writer.buffer.bytes (), o) switch {case O.status_code! = 0 && O.status_code < 20000:l .        Level = Level_error break case o.status_code > 20000:l.level = Level_warn break Default: L.level = Level_info break} l.respbody = L.writer.buffer.string () return L}func (l *log) Captureerro R (Err interface{}) *log {l.level = level_fatal switch Rval: = Err. ( Type) {Case error:l.respbody = rval.error () L.error = Rval break    Default:l.respbody = FMT. Sprint (rval) l.error = errors. New (l.respbody) Break} return L}func (L *log) Write (outwriter io. Writer) *log {l.timeuse = time. Now (). Sub (L.startat). String () Ojson, _: = json. Marshal (L) fmt. Fprintln (Outwriter, String (Ojson)) return L}func (L *log) report () {if l.level = Level_info | | l.level = LEVEL_WA RN {return} client: = Sentry. Client () client. Sethttpcontext (l.context.request) client. Setextracontext (map[string]interface{}{"Timeuse": l.timeuse}) Switch {case l.level = = level_fatal:client. Captureerror (L.level, l.error) break case l.level = = level_error:client. Capturemessage (L.level, l.respbody) Break}}

Because gin is a lightweight routing framework, there is no corresponding package for similar database operations and REDIS operations. This requires us to choose the right package.

Package-Database Operations

Datbase/sql was used in the initial learning phase, but the package has a very uncomfortable problem.

pid := 10021rows, err := db.Query("SELECT title FROM `product` WHERE id=?", pid)if err != nil {    log.Fatal(err)}defer rows.Close()for rows.Next() {    var title string    if err := rows.Scan(&title); err != nil {        log.Fatal(err)    }    fmt.Printf("%s is %d\n", title, pid)}if err := rows.Err(); err != nil {    log.Fatal(err)}

The code above, if the select is not the title, but *, then you need to advance all the fields in the table structure into a variable, and then passed to the scan method.

In this way, if there are more than 10 fields in a table, the development process can be extremely cumbersome. So what are we hoping for? Defining a field in advance is a must, but should normally be defined as a struct? What we expect is that query results can be converted directly into structured data.

Took a little time to find out, and finally found such a bag--github.com/jmoiron/sqlx.

    // You can also get a single result, a la QueryRow    jason = Person{}    err = db.Get(&jason, "SELECT * FROM person WHERE first_name=$1", "Jason")    fmt.Printf("%#v\n", jason)    // Person{FirstName:"Jason", LastName:"Moiron", Email:"jmoiron@jmoiron.net"}    // if you have null fields and use SELECT *, you must use sql.Null* in your struct    places := []Place{}    err = db.Select(&places, "SELECT * FROM place ORDER BY telcode ASC")    if err != nil {        fmt.Println(err)        return    }

SQLX is actually the expansion of the database/sql, so the development is not much more cool, Gaga ~

Why not use ORM? Or, as I said in the previous section, try not to package over-encapsulation.

Package-redis operation

Originally we used the Redigo "Github.com/garyburd/redigo/redis", but there is nothing wrong with the use of, but when the pressure test found a problem, that is, the use of the connection pool.

Func factory (name string) *redis. Pool {conf: = config. Get ("Redis." + name). (*TOML. Tomltree) Host: = Conf. Get ("host"). (string) Port: = Conf. Get ("Port"). (string) Password: = Conf. Getdefault ("passwd", ""). (string) fmt. Printf ("Conf-redis:%s:%s-%s\r\n", host, port, password) Pool: = &redis. pool{idletimeout:idletimeout, Maxidle:maxidle, Maxactive:maxactive, Dial:func () (Re Dis. Conn, error) {address: = FMT. Sprintf ("%s:%s", host, Port) C, err: = Redis. Dial ("TCP", Address, Redis.) Dialpassword (password),) if err! = Nil {exception. Catch (ERR) return nil, err} return C, nil},} return pool}/** * get connection * /func Getredis (name string) Redis. Conn {return redispool[name]. Get ()}/** * Get Master Connection */func Master (db int) redisclient {client: = redisclient{"Master", DB} return client}/** * received Take Slave connection */func Slave (db int) redisclient {client: = redisclient{"Slave", DB} return client} 

The above is defined as a connection pool, where a problem arises when the Redis command in Redigo is required to get the connection from the connection pool on its own, and it needs to put the connection back into the connection pool after use. Initially we did not put the connection back, leading to pressure measurement when the pressure is not up.

So there's no better package, the answer is certainly yes--gopkg.in/redis.v5

func factory(name string) *redis.Client {    conf := config.Get("redis." + name).(*toml.TomlTree)    host := conf.Get("host").(string)    port := conf.Get("port").(string)    password := conf.GetDefault("passwd", "").(string)    fmt.Printf("conf-redis: %s:%s - %s\r\n", host, port, password)    address := fmt.Sprintf("%s:%s", host, port)    return redis.NewClient(&redis.Options{        Addr:        address,        Password:    password,        DB:          0,        PoolSize:    maxActive,    })}/** * 获取连接 */func getRedis(name string) *redis.Client {    return factory(name)}/** * 获取master连接 */func Master() *redis.Client {    return getRedis("master")}/** * 获取slave连接 */func Slave() *redis.Client {    return getRedis("slave")}

As you can see, this package is returned directly to the required connection.

Then we go to see his source, the connection is not put back.

Func (c *baseclient) conn () (*pool. Conn, bool, error) {cn, isnew, Err: = C.connpool.get () if err! = Nil {return nil, false, err} if!CN . inited {if err: = C.initconn (CN); Err! = Nil {_ = C.connpool.remove (cn, err) return nil, FA LSE, ERR}} return cn, IsNew, Nil}func (c *baseclient) Putconn (CN *pool. Conn, err error, allowtimeout bool) bool {if internal.    Isbadconn (Err, allowtimeout) {_ = C.connpool.remove (cn, err) return false} _ = C.connpool.put (CN)        Return True}func (c *baseclient) defaultprocess (cmd Cmder) error {for i: = 0; I <= c.opt.maxretries; i++ { CN, _, Err: = C.conn () if err! = Nil {Cmd.seterr (err) return Err} CN. Setwritetimeout (c.opt.writetimeout) If err: = Writecmd (cn, cmd); Err! = Nil {C.putconn (CN, err, False) Cmd.seterr (ERR) if err! = Nil && Internal. Isretryableerror (Err) {               Continue} return err} CN. Setreadtimeout (c.cmdtimeout (cmd)) Err = Cmd.readreply (CN) C.putconn (CN, err, FALSE) if err! = Nil &am p;& Internal. Isretryableerror (Err) {continue} return err} return cmd. ERR ()}

As you can see, the underlying operation in this package first goes to get a connection in Connpool, and then executes the Putconn method to put the connection back to Connpool.

Conclusion

package mainimport (    "github.com/gin-gonic/gin"    "gosapi/application/library/initd"    "gosapi/application/routers")func main() {    env := initd.ConfTree.Get("ENVIRONMENT").(string)    gin.SetMode(env)    router := gin.New()    routers.Register(router)    router.Run(":7321") // listen and serve on 0.0.0.0:7321}

March 21 began to write main, now has been online for one weeks, has not found any problems.

The performance has been improved by about four times times after the pressure measurement. The original response time was around 70 milliseconds, and it is now around 10 milliseconds. The original throughput was about 1200, and now it's about 3300.

Although go is great, but I still want to say: PHP is the best language!