Gin Practice Serial 12 optimize configuration structure and implement image upload

This is a creation in Article, where the information may have evolved or changed.

Optimize configuration structure and implement image upload

Project Address: Https://github.com/EDDYCJY/go ...

If it helps, you're welcome to a Star.

Objective

One day, the product manager suddenly told you the article list, no cover map, not beautiful enough! ) &￥! &) #&￥! Add one, for a few minutes.

You open your program and analyze a wave to write a list:

• Optimize the configuration structure (because of the increasing number of configuration items)
• Extracting the File from the original logging is convenient for public (logging, upload each one is not appropriate)
• Implement upload Image interface (need to limit file format, size)
• Modify the article interface (need to support the cover page address parameter)
• Add database fields for Blog_article (article)
• Implement HTTP. Fileserver

Well, if you find it better, you need to tweak some of the application structure, because there are more functions and the original design is going to keep pace.

That is, in due time, optimize

Optimize configuration structure

First, explain

In the previous chapters, the use of direct read key to store configuration items, and the need to increase the picture configuration items, the overall is a bit redundant

We use the following workaround:

• Map structure: Use Mapto to set configuration parameters
• Configuration: All configuration items are in setting

Map structure Body (example)

In Go-ini, you can map structs in a mapto way, for example:

type Server struct {    RunMode string    HttpPort int    ReadTimeout time.Duration    WriteTimeout time.Duration}var ServerSetting = &Server{}func main() {    Cfg, err := ini.Load("conf/app.ini")    if err != nil {        log.Fatalf("Fail to parse 'conf/app.ini': %v", err)    }        err = Cfg.Section("server").MapTo(ServerSetting)    if err != nil {        log.Fatalf("Cfg.MapTo ServerSetting err: %v", err)    }}

In this code, you can note that serversetting took the address, why Mapto must address the parameter?

// MapTo maps section to given struct.func (s *Section) MapTo(v interface{}) error {    typ := reflect.TypeOf(v)    val := reflect.ValueOf(v)    if typ.Kind() == reflect.Ptr {        typ = typ.Elem()        val = val.Elem()    } else {        return errors.New("cannot map to non-pointer struct")    }    return s.mapTo(val, false)}

An error that typ.Kind() == reflect.Ptr is bound to be returned if a pointer must be used in Mapto cannot map to non-pointer struct . This is a superficial reason.

More inward probing, which can be thought field.Set of as the cause, when executed val := reflect.ValueOf(v) , the function is created by passing the v copy, but the change does not change the original, and the change that is to be made val val v val will have to be v passed v The address

Obviously Go-ini also contains the function of modifying the original value, what do you think is the reason?

Configuration of the governs

In previous versions, models and file configurations were parsed in their own files, while others were in setting.go, so we needed to unify them in setting

You might want to copy and paste the configuration items directly into Setting.go's init, and you're done.

But you're thinking, there's more than one init function in the previous code, there's a problem with the order of execution, and you can't meet our requirements, you can try

(Here is a basic point of knowledge)

In Go, when there are multiple init functions, the Order of execution is:

• init functions under the same package: order of execution by source file compilation (default sort by file name)
• init functions under different packages: determining the order of dependencies by package import

So to avoid multiple init situations, try to initialize the sequence by the program control

Second, the implementation

Modifying a configuration file

Open Conf/app.ini To modify the configuration file to the big hump name, in addition we added 5 configuration items for uploading the image function, 4 file log aspects of the configuration items

[app]PageSize = 10JwtSecret = 233RuntimeRootPath = runtime/ImagePrefixUrl = http://127.0.0.1:8000ImageSavePath = upload/images/# MBImageMaxSize = 5ImageAllowExts = .jpg,.jpeg,.pngLogSavePath = logs/LogSaveName = logLogFileExt = logTimeFormat = 20060102[server]#debug or releaseRunMode = debugHttpPort = 8000ReadTimeout = 60WriteTimeout = 60[database]Type = mysqlUser = rootPassword = rootrootHost = 127.0.0.1:3306Name = blogTablePrefix = blog_

Optimized configuration read and set initialization order

The first step

Delete the configuration that is scattered in other files, unify the processing in setting , and Modify the init function as the Setup method

Open the Pkg/setting/setting.go file and modify the following:

Package Settingimport ("Log", "Time" "Github.com/go-ini/ini") type App struct {jwtsecret string PageSize in T Runtimerootpath string Imageprefixurl string Imagesavepath string imagemaxsize int imageallowexts []strin G Logsavepath String Logsavename string Logfileext string timeformat string}var appsetting = &app{}type Se RVer struct {runmode string httpport int readtimeout time. Duration writetimeout time. Duration}var serversetting = &server{}type Database struct {type string User string Password string Host String Name string tableprefix string}var databasesetting = &database{}func Setup () {CFG, err: = ini. Load ("Conf/app.ini") if err! = Nil {log. Fatalf ("Fail to Parse ' Conf/app.ini ':%v", err)} err = Cfg.section ("app"). Mapto (appsetting) if err! = Nil {log.    Fatalf ("Cfg.mapto appsetting err:%v", err)} appsetting.imagemaxsize = appsetting.imagemaxsize * 1024 * 1024Err = cfg.section ("server"). Mapto (serversetting) if err! = Nil {log. Fatalf ("Cfg.mapto serversetting err:%v", err)} serversetting.readtimeout = Serversetting.readtimeout * time. Second serversetting.writetimeout = serversetting.readtimeout * time. Second err = cfg.section ("database"). Mapto (databasesetting) if err! = Nil {log. Fatalf ("Cfg.mapto databasesetting err:%v", Err)}}

Here are a few things we did:

• Write structures that are consistent with configuration items (App, Server, Database)
• To map a configuration item to a struct using mapto
• To re-assign some configuration items that require special settings
• Modify the init function of Models.go, Setting.go, Pkg/logging/log.go to the Setup method
• Delete the Models/models.go independently read DB configuration item and read the setting
• Remove the Pkg/logging/file separate LOG configuration item and read it uniformly setting

After a few comparison basis, and did not post out, I believe you can solve, if there is a problem, you can turn right project address

Step Two

In this step we will set the initialization process, open the Main.go file, and modify the content:

func main() {    setting.Setup()    models.Setup()    logging.Setup()    endless.DefaultReadTimeOut = setting.ServerSetting.ReadTimeout    endless.DefaultWriteTimeOut = setting.ServerSetting.WriteTimeout    endless.DefaultMaxHeaderBytes = 1 << 20    endPoint := fmt.Sprintf(":%d", setting.ServerSetting.HttpPort)    server := endless.NewServer(endPoint, routers.InitRouter())    server.BeforeBegin = func(add string) {        log.Printf("Actual pid is %d", syscall.Getpid())    }    err := server.ListenAndServe()    if err != nil {        log.Printf("Server err: %v", err)    }}

Once the modification is complete, the initialization function of the multi-module is successfully put into the initiation process (it can be controlled in order)

Verify

So far, the configuration optimization for this requirement is complete, and you need to go run main.go verify that your functionality is working properly.

Incidentally, let's leave a basic question and we can think about it.

ServerSetting.ReadTimeout = ServerSetting.ReadTimeout * time.SecondServerSetting.WriteTimeout = ServerSetting.ReadTimeout * time.Second

What happens if you delete these two lines in the Setting.go file, and what is the problem?

Pulling away from File

In previous versions, we used some of the OS methods in Logging/file.go, which we discovered through pre-planning, which can be reused in the upload image function.

The first step

Create a new file/file.go in the PKG directory and write the following file:

package fileimport (    "os"    "path"    "mime/multipart"    "io/ioutil")func GetSize(f multipart.File) (int, error) {    content, err := ioutil.ReadAll(f)    return len(content), err}func GetExt(fileName string) string {    return path.Ext(fileName)}func CheckExist(src string) bool {    _, err := os.Stat(src)    return os.IsNotExist(err)}func CheckPermission(src string) bool {    _, err := os.Stat(src)    return os.IsPermission(err)}func IsNotExistMkDir(src string) error {    if exist := CheckExist(src); exist == false {        if err := MkDir(src); err != nil {            return err        }    }    return nil}func MkDir(src string) error {    err := os.MkdirAll(src, os.ModePerm)    if err != nil {        return err    }    return nil}func Open(name string, flag int, perm os.FileMode) (*os.File, error) {    f, err := os.OpenFile(name, flag, perm)    if err != nil {        return nil, err    }    return f, nil}

Here we have a total of 7 methods encapsulated

• GetSize: Get File size
• Getext: Get file suffix
• Checkexist: Check if the file exists
• Checkpermission: Check file permissions
• Isnotexistmkdir: Create a new folder if it does not exist
• MkDir: New Folder
• Open: Opening file

Here we use the mime/multipart package, it mainly implements the MIME multipart parsing, mainly for HTTP and common browser-generated multipart body

Multipart is what, rfc2388 Multipart/form-data understand

Step Two

We have re-encapsulated the file in the first step, and in this step we will fix the original logging package.

1. Open the Pkg/logging/file.go file and modify the contents of the file:

Package Loggingimport ("FMT" "OS" "Time" "github.com/eddycjy/go-gin-example/pkg/setting" "Github.com/eddyc Jy/go-gin-example/pkg/file ") func Getlogfilepath () string {return FMT. Sprintf ("%s%s", setting. Appsetting.runtimerootpath, setting. Appsetting.logsavepath)}func getlogfilename () string {return FMT. Sprintf ("%s%s.%s", setting. Appsetting.logsavename, time. Now (). Format (setting. Appsetting.timeformat), setting. Appsetting.logfileext,)}func openlogfile (FileName, FilePath string) (*os. File, error) {dir, err: = OS. GETWD () if err! = Nil {return nil, fmt. Errorf ("OS. GETWD err:%v ", err)} src: = dir +"/"+ filePath perm: = file. Checkpermission (src) if perm = = true {return nil, fmt. Errorf ("file. Checkpermission Permission denied src:%s ", src)} err = file. Isnotexistmkdir (SRC) if err! = Nil {return nil, fmt. Errorf ("file. Isnotexistmkdir src:%s, err:%v ", SRC, err)} F, err: = file. Open (src + FIlename, OS. O_append|os. O_create|os. O_wronly, 0644) if err! = Nil {return nil, fmt. Errorf ("Fail to OpenFile:%v", err)} return F, nil}

We're going to change the quote to the method in the File/file.go package.

2. Open the Pkg/logging/log.go file and modify the contents of the file:

package logging...func Setup() {    var err error    filePath := getLogFilePath()    fileName := getLogFileName()    F, err = openLogFile(fileName, filePath)    if err != nil {        log.Fatalln(err)    }    logger = log.New(F, DefaultPrefix, log.LstdFlags)}...

Since the original method parameters have changed, the OpenLogFile also needs to be adjusted

Implement upload Image interface

This section, we begin to implement some of the previous image-related methods and functions

You first need to add a field to the blog_article in the cover_image_url formatvarchar(255) DEFAULT '' COMMENT '封面图片地址'

0th Step

Generally does not directly expose the uploaded image name, so we MD5 the image name to achieve this effect

Create a new md5.go in the Util directory and write the contents of the file:

package utilimport (    "crypto/md5"    "encoding/hex")func EncodeMD5(value string) string {    m := md5.New()    m.Write([]byte(value))    return hex.EncodeToString(m.Sum(nil))}

The first step

We've already encapsulated the underlying approach in the first place, which is the process logic for encapsulating image

Create a new Upload/image.go file in the pkg directory and write the contents of the file:

Package Uploadimport ("OS" "Path" "Log" "FMT" "Strings" "Mime/multipart" "github.com/eddycjy/go-gin- Example/pkg/file "" Github.com/eddycjy/go-gin-example/pkg/setting "" github.com/eddycjy/go-gin-example/pkg/logging "Github.com/eddycjy/go-gin-example/pkg/util") Func Getimagefullurl (name string) string {return setting. Appsetting.imageprefixurl + "/" + Getimagepath () + Name}func getimagename (name string) String {ext: = path. EXT (name) FileName: = Strings. Trimsuffix (name, ext) fileName = util. EncodeMD5 (filename) return filename + ext}func Getimagepath () string {return setting. Appsetting.imagesavepath}func Getimagefullpath () string {return setting. Appsetting.runtimerootpath + getimagepath ()}func checkimageext (FileName string) bool {ext: = file. Getext (FileName) for _, Allowext: = Range setting. appsetting.imageallowexts {if strings. ToUpper (allowext) = = Strings. ToUpper (EXT) {return true}} return FalsE}func checkimagesize (f multipart. File) bool {size, err: = file. GetSize (f) if err! = Nil {log. PRINTLN (Err) logging. Warn (ERR) return false} return size <= setting. Appsetting.imagemaxsize}func checkimage (src string) error {dir, err: = OS. GETWD () if err! = Nil {return FMT. Errorf ("OS. GETWD err:%v ", err)} err = file. Isnotexistmkdir (dir + "/" + src) if err! = Nil {return FMT. Errorf ("file. Isnotexistmkdir err:%v ", Err)} perm: = file. Checkpermission (src) if perm = = true {return FMT. Errorf ("file. Checkpermission Permission denied src:%s ", SRC)} return nil}

Here we have implemented 7 methods, as follows:

• Getimagefullurl: Get picture full access URL
• Getimagename: Get Picture name
• Getimagepath: Get Picture path
• Getimagefullpath: Get picture full path
• Checkimageext: Check image suffix
• Checkimagesize: Check the image size
• Checkimage: Checking pictures

This is basically two times the underlying code package, in order to more flexible processing some of the image-specific logic, and easy to modify, not directly exposed to the lower layer

Step Two

This step will write the business logic to upload the picture, create a new Upload.go file in the Routers/api directory, write the contents of the file:

Package Apiimport ("Net/http" "Github.com/gin-gonic/gin" "github.com/eddycjy/go-gin-example/pkg/e" "github.co M/eddycjy/go-gin-example/pkg/logging "" Github.com/eddycjy/go-gin-example/pkg/upload ") func UploadImage (c *gin. Context) {code: = e.success Data: = Make (map[string]string) file, image, Err: = C.request.formfile ("image") I F Err! = Nil {logging. Warn (ERR) code = E.error C.json (http. Statusok, Gin. h{"Code": Code, "MSG": E.getmsg (Code), "Data": Data,})} if image = = Nil {code = E.invalid_params} else {imageName: = upload. Getimagename (image. Filename) FullPath: = Upload. Getimagefullpath () Savepath: = Upload. Getimagepath () src: = FullPath + imageName if! Upload. Checkimageext (imageName) | | ! Upload. Checkimagesize (file) {code = E.error_upload_check_image_format} else {err: = UPLOAD. Checkimage (FullPath) IF Err! = Nil {logging. Warn (ERR) code = E.error_upload_check_image_fail} else If err: = C.saveuploadedfile (IMAGE, SRC) ; Err! = Nil {logging. Warn (ERR) code = E.error_upload_save_image_fail} else {data["image_url"] = Uplo Ad. Getimagefullurl (imageName) data["image_save_url"] = Savepath + ImageName}}} c.js On (HTTP. Statusok, Gin. h{"Code": Code, "MSG": E.getmsg (Code), "Data": Data,})}

The error code involved (to be added in Pkg/e/code.go, msg.go):

// 保存图片失败ERROR_UPLOAD_SAVE_IMAGE_FAIL = 30001// 检查图片失败ERROR_UPLOAD_CHECK_IMAGE_FAIL = 30002// 校验图片错误，图片格式或大小有问题ERROR_UPLOAD_CHECK_IMAGE_FORMAT = 30003

In this large segment of business logic, we do the following things:

• C.request.formfile: Gets the uploaded picture (the first file that returns the provided table one-touch)
• Checkimageext, checkimagesize Check the image size, check the image suffix
• Checkimage: Check the upload image required (permissions, folders)
• Saveuploadedfile: Save Picture

In general, it is the application process of the check-save

Step Three

Open the Routers/router.go file to add routes r.POST("/upload", api.UploadImage) such as:

func InitRouter() *gin.Engine {    r := gin.New()    ...    r.GET("/auth", api.GetAuth)    r.GET("/swagger/*any", ginSwagger.WrapHandler(swaggerFiles.Handler))    r.POST("/upload", api.UploadImage)    apiv1 := r.Group("/api/v1")    apiv1.Use(jwt.JWT())    {        ...    }    return r}

Verify

Finally, we request to upload the image interface, test the function written

Check if the directory contains files (attention to permissions issues)

$ pwd$GOPATH/src/github.com/EDDYCJY/go-gin-example/runtime/upload/images$ ll... 96a3be3cf272e017046d1b2674a52bd3.jpg... c39fa784216313cf2faa7c98739fc367.jpeg

Here we have returned a total of 2 parameters, one is the full access URL, the other is to save the path

Implement HTTP. Fileserver

After we have completed the previous section, we also need to have the front-end access to the picture, which is generally as follows:

• Cdn
• http. FileSystem

In the company, CDN or self-built distributed file system is mostly, do not need too much attention. In practice, the words must be built locally, Go itself has a good support for this, and Gin is again encapsulated a layer, only need to add a line of code in the route can be

Open the Routers/router.go file to add routes r.StaticFS("/upload/images", http.Dir(upload.GetImageFullPath())) such as:

func InitRouter() *gin.Engine {    ...    r.StaticFS("/upload/images", http.Dir(upload.GetImageFullPath()))    r.GET("/auth", api.GetAuth)    r.GET("/swagger/*any", ginSwagger.WrapHandler(swaggerFiles.Handler))    r.POST("/upload", api.UploadImage)    ...}

When the $HOST/upload/images is accessed, the files under $GOPATH/src/github.com/eddycjy/go-gin-example/runtime/upload/images will be read

And what does this line of code do, let's look at the method prototype

// StaticFS works just like `Static()` but a custom `http.FileSystem` can be used instead.// Gin by default user: gin.Dir()func (group *RouterGroup) StaticFS(relativePath string, fs http.FileSystem) IRoutes {    if strings.Contains(relativePath, ":") || strings.Contains(relativePath, "*") {        panic("URL parameters can not be used when serving a static folder")    }    handler := group.createStaticHandler(relativePath, fs)    urlPattern := path.Join(relativePath, "/*filepath")    // Register GET and HEAD handlers    group.GET(urlPattern, handler)    group.HEAD(urlPattern, handler)    return group.returnObj()}

First, the use of * and: symbols is forbidden in the exposed URL, and by createStaticHandler creating a static file service, the actual final call or fileServer.ServeHTTP some processing logic

Below you can see urlPattern := path.Join(relativePath, "/*filepath") , /*filepath who you are, what you use here, you are the product of Gin?

The semantics are known to be the processing logic of the route, and the Gin route is based on the Httprouter, and the following information can be obtained by consulting the document.

Pattern: /src/*filepath /src/                     match /src/somefile.go          match /src/subdir/somefile.go   match

filepath will match all file paths, and filepath must be at the end of the Pattern

Verify

Re go run main.go -Execute, go to access the image address just obtained on the upload interface, check http. FileSystem is normal

Modify the article Interface

Next, you need to modify Routers/api/v1/article.go's addarticle, editarticle two interfaces

• New, updated article interface: support for the entry Cover_image_url
• New, updated article interface: increased non-empty, maximum length check for Cover_image_url

The previous article says that if you have a question, you can refer to the code of the project

Summarize

In this chapter, we briefly analyze the requirements, make a small plan for the application and implement

Complete the function points and optimizations in the checklist, which is a common scenario in the actual project, and I hope you can savor them and learn more about them.

Reference

Sample code for this series

• Go-gin-example

Catalog of this series

• Gin Practice Serial One Golang introduction and environment installation
• Gin Practice Two Build blog API ' s (i)
• Gin Practice three Build blog API ' s (ii)
• Gin Practice Four Build blog API ' s (iii)
• Gin Practice Five Build blog API ' s (iv)
• Gin Practice Six Build blog API ' s (v)
• Gin Practice serial Seven Golang graceful restart HTTP service
• Gin practice serial Eight for it plus swagger
• Gin practice nine deploying Golang applications to Docker
• Gin Practice serial 10 Custom GORM callbacks
• Gin Practice Serial 11 Cron timed tasks
• Gin Practice Golang Cross-compiling