Summary of Golang program configuration scheme

This is a creation in Article, where the information may have evolved or changed.

See a series of articles on Twitter about the Golang program configuration summary (a mini series, total 6 articles), with the original link: here. I feel good, here rough finishing (not full-text translation), for your reference.

I. BACKGROUND

No matter how you develop your application in any programming language, you can rely on configuration data. Configuration data is available in a variety of forms, including command-line options, parameters (parameters), environment variables (env VARs), and configuration files. Golang is no exception. Golang built-in Flag standard library, can be used to support the resolution of some command line options and parameters, Golang through the OS package provides methods to obtain the current environment variables, but Golang does not specify the standard profile format (although the built-in support XML, JSON), more through third-party Package to resolve the problem with the configuration file read. Golang configuration related to the third-party mail a lot, the author in this article is given in the configuration scheme contains the mainstream third-party configuration data action package.

The author believes that a good application configuration hierarchy should be this:
1. Initial default values for built-in configuration items within the program
2. The configuration item value in the configuration file can override the default value of the configuration item within the program (override).
3. Command-line options and parameter values have the highest precedence and override the first two levels of the configuration item values.

The following is the author's thoughts on a gradual approach to Golang program configuration scheme.

II. parsing command-line options and parameters

This section focuses on how the Golang program accesses command-line options and parameters.

Golang provides built-in support for access to command-line arguments:

Cmdlineargs.go
Package Main

Import (
"FMT"
"OS"
"Path/filepath"
)

Func Main () {
println ("I AM", os. Args[0])

BaseName: = filepath. Base (OS. Args[0])
println ("The base name is", BaseName)

The length of array A can be discovered using the built-in function len
println ("Argument # is", Len (OS). Args))

The first command line arguments
If Len (OS. Args) > 1 {
println ("The first command line argument:", OS.) ARGS[1])
}
}

The results of the implementation are as follows:
$go Build Cmdlineargs.go
$cmdlineargs Test One
I am Cmdlineargs
The base name is Cmdlineargs
Argument # is 3
The first command line Argument:test

For programs with complex command-line structures, we will at least use the flag package built into the Golang standard library:

Cmdlineflag.go
Package Main

Import (
"Flag"
"FMT"
"OS"
"StrConv"
)

VAR (
Main operation modes
Write = flag. Bool ("W", False, "write result back instead of Stdout\n\t\tdefault:no write back")

Layout control
Tabwidth = flag. Int ("Tabwidth", 8, "tab Width\n\t\tdefault:standard")

Debugging
Cpuprofile = flag. String ("Cpuprofile", "", "Write CPU profile to this file\n\t\tdefault:no default")
)

Func usage () {
fprintf allows us to print to a specifed file handle or stream
Fmt. fprintf (OS. Stderr, "\nusage:%s [flags] file [path ...] \ n ",
"Commandlineflag")//Os. Args[0]
Flag. Printdefaults ()
Os. Exit (0)
}

Func Main () {
Fmt. Printf ("Before parsing the flags\n")
Fmt. Printf ("T:%d\nw:%s\nc: '%s ' \ n",
*tabwidth, StrConv. Formatbool (*write), *cpuprofile)

Flag. Usage = Usage
Flag. Parse ()

There is also a mandatory non-flag arguments
If Len (flag. Args ()) < 1 {
Usage ()
}

Fmt. PRINTF ("Testing the Flag package\n")
Fmt. Printf ("T:%d\nw:%s\nc: '%s ' \ n",
*tabwidth, StrConv. Formatbool (*write), *cpuprofile)

For index, element: = Range flag. Args () {
Fmt. Printf ("I:%d C: '%s ' \ n", index, Element)
}
}

In this example:
-Describes the usage of three types of flags: Int, String, and bool.
-explains that the definition of each flag consists of type, command-line option text, default value, and meaning interpretation.
-finally explains how to handle the flag option and the non-option parameter.

Run with no parameters:

$cmdlineflag
Before parsing the flags
T:8
W:false
C: "

Usage:commandlineflag [flags] file [path ...]

-cpuprofile= "": Write CPU profile to the This file
Default:no Default
-tabwidth=8:tab width
Default:standard
-w=false:write result back instead of stdout
Default:no Write back

Run with command line flags and parameters (one without flag, one with two flags):

$cmdlineflag AA bb
Before parsing the flags
T:8
W:false
C: "
Testing the Flag Package
T:8
W:false
C: "
i:0 C: ' AA '
I:1 C: ' BB '

$cmdlineflag-tabwidth=2-w AA
Before parsing the flags
T:8
W:false
C: "
Testing the Flag Package
T:2
W:true
C: "
i:0 C: ' AA '

As you can see from the example, in a simple case, you don't have to write your own command line parser or use a third-party package, and using the go built-in Flag pack is a good way to get the job done. But the Golang flag package and the fact standard of command line parser: Posix getopt (C/c++/perl/shell scripts are available) compared to the larger gap, mainly reflected in:

1. Cannot support distinguishing long option from short option, for example:-H and help.
2, does not support the short options merger, for example: Ls-l-H <=> ls-hl
3, the location of the command line flag can not be arbitrarily placed, such as can not be placed behind the Non-flag parameter.

But after all, flag is golang built-in standard library package, you can use it without any cost. The addition of the bool-type flag is also one of its major highlights.

The fact standard of Toml,go configuration file (this may not be recognized)

Although the command line is an optional configuration scenario, more often than not, we use configuration files to store static configuration data. Just like Java with Xml,ruby and yaml,windows with Ini,go also has its own combination, that is toml (Tom's obvious, Minimal Language).

At first glance TOML syntax is similar to Windows INI, but careful research you will find it far more powerful than INI, here is an example of a toml configuration file:

# This is a TOML document. Boom.

title = "TOML Example"

[owner]
Name = "Lance Uppercut"
DOB = 1979-05-27t07:32:00-08:00 # First class dates? Why isn't?

[Database]
Server = "192.168.1.1"
Ports = [8001, 8001, 8002]
Connection_max = 5000
Enabled = True

[Servers]

# you can indent as your please. Tabs or spaces. TOML don ' t care.
[Servers.alpha]
ip = "10.0.0.1"
DC = "EQDC10"

[Servers.beta]
ip = "10.0.0.2"
DC = "EQDC10"

[Clients]
data = [["Gamma", "Delta"], [1, 2]]

# line breaks is OK when inside arrays
hosts = [
"Alpha",
"Omega"
]

It looks very powerful and complex, but it's easy to parse. Take the following toml file as an example:

Age = 25
Cats = ["Cauchy", "Plato"]
Pi = 3.14
perfection = [6, 28, 496, 8128]
DOB = 1987-07-05t05:45:00z

Similar to all other configuration file parser, the data in this configuration file can be parsed directly into a golang struct:

Type Config struct {
Age int
Cats []string
Pi float64
perfection []int
DOB time. Time//requires ' import time '
}

The steps for parsing are simple:

var conf Config
If _, Err: = toml. Decode (Tomldata, &conf); Err! = Nil {
Handle error
}

is not simple can not be simple!

But TOML also has its shortcomings. What should you do if you need to override the options in these configuration files using the parameter values of the command line options? In fact, we often encounter situations like the following three-tier configuration structure:

1. Initial default values for built-in configuration items within the program
2. The configuration item value in the configuration file can override the default value of the configuration item within the program (override).
3. Command-line options and parameter values have the highest precedence and override the first two levels of the configuration item values.

In go, the result body field of the TOML map has no initial value. And the go built-in flag package does not parse the command-line parameter values into a go structure, but rather a fragmented variable. These can be solved through third-party tools, but if you don't want to use third-party tools, you can do it yourself, though it's ugly.

Func configget () *config {
var err error
var cf *config = Newconfig ()

Set default values defined in the program
Cf. Configfromflag ()
Log. Printf ("P:%d, B: '%s ', F: '%s ' \ n", cf.) Maxprocs, cf. Webapp.path)

Load config file, from flag or env (if specified)
_, err = cf. Configfromfile (*configfile, OS. Getenv ("APPCONFIG"))
If err! = Nil {
Log. Fatal (ERR)
}
Log. Printf ("P:%d, B: '%s ', F: '%s ' \ n", cf.) Maxprocs, cf. Webapp.path)

Override values from command line flags
Cf. Configtoflag ()
Flag. Usage = Usage
Flag. Parse ()
Cf. Configfromflag ()
Log. Printf ("P:%d, B: '%s ', F: '%s ' \ n", cf.) Maxprocs, cf. Webapp.path)

Cf. Configapply ()

Return CF
}

Just like in the code above, you need to:
1. Set the configuration (CF) default value with the command line flag default value.
2. Load the configuration file next
3. Overwrite the command line flag variable value with the configuration value (CF)
4. Parsing command Line parameters
5. Override the configuration (CF) value with the command line flag variable value.

You won't be able to achieve three-tier configuration with one less step.

Iv. Beyond TOML

This section will focus on how to overcome the limitations of toml.

In order to achieve this goal, many people will say: using Viper, but before introducing Viper this heavyweight player, I would like to introduce another not so well-known players: Multiconfig.

Some people always think that the big one is good, but I believe the fit is better. Because:

1, Viper too heavyweight, when using Viper you need to pull another 20 Viper dependent third-party packages
2, in fact, Viper alone is not enough to meet the demand, to want to get viper all functions, you also need another package, the latter is dependent on 13 external packages
3, compared with Viper, Multiconfig is simpler to use.

Well, let's look back at the problems we're facing:

1, define the default configuration in the program, so that we no longer have to define them in the TOML.
2. Override the default configuration with data from the TOML configuration file
3. Override the configuration read from TOML with the value of the command line or environment variable.

Here's an example that shows how to use Multiconfig:

Func Main () {
M: = Multiconfig. Newwithpath ("config.toml")//supports TOML and JSON

Get an empty struct for your configuration
serverconf: = new (Server)

Populated the serverconf struct
M.mustload (serverconf)//Check for error

Fmt. Println ("After Loading:")
Fmt. Printf ("%+v\n", serverconf)

If serverconf.enabled {
Fmt. Println ("Enabled field is set to true")
} else {
Fmt. Println ("Enabled field is set to false")
}
}

The toml file in this example is as follows:

Name = "Koding"
Enabled = False
Port = 6066
Users = ["Ankara", "Istanbul"]

[Postgres]
Enabled = True
Port = 5432
Hosts = ["192.168.2.1", "192.168.2.2", "192.168.2.3"]
Availabilityratio = 8.23

The go structure after TOML mapping is as follows:

Type (
Server holds supported types by the Multiconfig package
Server struct {
Name string
Port int ' default: ' 6060 '
Enabled BOOL
Users []string
Postgres Postgres
}

Postgres is here for embedded struct feature
Postgres struct {
Enabled BOOL
Port int
Hosts []string
DBName string
Availabilityratio float64
}
)

The use of Multiconfig is not very simple, after the follow-up with the Viper, you will agree with my point of view.

Multiconfig supports the default values and also supports explicit field assignment requirements.
Supports TOML, JSON, struct tags (struct tags), and environment variables.
You can customize the configuration source (such as a remote server) if you want to do so.
can be highly extended (via the loader interface), you can create your own loader.

Here is the result of the example running, first of all the usage help:

$cmdlinemulticonfig-help
Usage of Cmdlinemulticonfig:
 -enabled=false:change value of enabled.
 -name=koding:change value of name.
 -port=6066:change value of port.
 -postgres-availabilityratio=8.23:change value of Postgres-availabilityratio.
 -postgres-dbname=: Change value of Postgres-dbname.
 -postgres-enabled=true:change value of postgres-enabled.
 -postgres-hosts=[192.168.2.1 192.168.2.2 192.168.2.3]: Change value of postgres-hosts.
 -postgres-port=5432:change value of Postgres-port.
 -users=[ankara Istanbul]: Change value of users.

Generated Environment variables:
server_name
Server_port
Server_enabled
Server_users
Server_postgres_enabled
Server_postgres_port
Server_postgres_hosts
Server_postgres_dbname
Server_postgres_availabilityratio

$cmdlinemulticonfig
After Loading:
&{name:koding port:6066 enabled:false Users:[ankara Istanbul] Postgres:{enabled:true port:5432 Hosts:[192.168.2.1 192.168.2.2 192.168.2.3] dbname:availabilityratio:8.23}}
Enabled field is set to False

Check the output, is not every match our previous expectations!

Wu, Viper

Our heavyweight viper (Https://github.com/spf13/viper) is on the stage!

There is no doubt that Viper is very powerful. But if you want to overwrite a predefined configuration item value with a command-line parameter, viper yourself is not enough. To make Viper explode, you need another package, which is Cobra (Https://github.com/spf13/cobra).

Unlike multiconfig,viper that focus on simplifying configuration processing, you have total control. Unfortunately, you need to do some physical work before you get this kind of control.

Let's take a look back at the code that uses the Multiconfig processing configuration:

Func Main () {
M: = Multiconfig. Newwithpath ("config.toml")//supports TOML and JSON

Get an empty struct for your configuration
serverconf: = new (Server)

Populated the serverconf struct
M.mustload (serverconf)//Check for error

Fmt. Println ("After Loading:")
Fmt. Printf ("%+v\n", serverconf)

If serverconf.enabled {
Fmt. Println ("Enabled field is set to true")
} else {
Fmt. Println ("Enabled field is set to false")
}
}

That's all you have to do when you use Multiconfig. Now let's take a look at how to do the same thing with Viper and Cobra:

Func init () {
Maincmd.addcommand (Versioncmd)

Viper. Setenvprefix ("DISPATCH")
Viper. Automaticenv ()

/*
When Automaticenv called, Viper would check for a environment variable any
Time a viper. Get request is made. It'll apply the following rules. It
Would check for a environment variable with a name matching the key
Uppercased and prefixed with the envprefix if set.
*/

Flags: = Maincmd.flags ()

    flags. Bool ("Debug", False, "Turn on Debugging.")
    flags. String ("addr", "localhost:5002", "Address of the Service")
    flags. String ("Smtp-addr", "localhost:25", "Address of the SMTP server")
    flags. String ("Smtp-user", "", "user to authenticate with the SMTP server")
    flags. String ("Smtp-password", "", "password to authenticate with the SMTP server")
    flags. String ("Email-from", "noreply@example.com", "the From email address.")

Viper. Bindpflag ("Debug", Flags.) Lookup ("Debug"))
Viper. Bindpflag ("addr", flags. Lookup ("addr"))
Viper. Bindpflag ("smtp_addr", flags. Lookup ("Smtp-addr"))
Viper. Bindpflag ("Smtp_user", flags. Lookup ("Smtp-user"))
Viper. Bindpflag ("Smtp_password", flags. Lookup ("Smtp-password"))
Viper. Bindpflag ("Email_from", flags. Lookup ("Email-from"))

Viper supports reading from YAML, TOML and/or JSON files. Viper can
Search multiple paths. Paths'll be searched in the order they is
provided. Searches stopped once Config File found.

Viper. Setconfigname ("COMMANDLINECV")//Name of config file (without extension)
Viper. Addconfigpath ("/tmp")//path to look for the config file in
Viper.             Addconfigpath (".") More path-to-look for the config files

ERR: = Viper. Readinconfig ()
If err! = Nil {
println ("No config file found. Using built-in defaults. ")
}
}

As you can see, you need to use Bindpflag to make Viper and cobra work together. But that's not too bad.

The real power of Cobra is its ability to provide subcommand. Cobra also provides a full range of POSIX-compatible command-line flag parsing capabilities, including long and short flags, inline commands, defining your own Help or usage for command.

The following is an example code that defines a subcommand:

The main command describes the service and defaults to printing the
Help message.
var maincmd = &cobra.command{
Use: "Dispatch",
Short: "Event dispatch service.",
Long: ' HTTP service that consumes events and dispatches them to subscribers. ',
Run:func (cmd *cobra.command, args []string) {
Serve ()
},
}

The version command prints this service.
var versioncmd = &cobra.command{
Use: "Version",
Short: "Print the version.",
Long: "The version of the Dispatch service.",
Run:func (cmd *cobra.command, args []string) {
Fmt. PRINTLN (Version)
},
}

With the definition of subcommand above, we can get the following help information:

Usage:
dispatch [Flags]
Dispatch [command]

Available Commands:
Version Print the version.
Help Help on any command

Flags:
–addr= "localhost:5002": Address of the service
–debug=false:turn on debugging.
–email-from= "noreply@example.com": the From email address.
-h,–help=false:help for Dispatch
–smtp-addr= "localhost:25": Address of the SMTP server
–smtp-password= "": Password to authenticate with the SMTP server
–smtp-user= "": User to authenticate with the SMTP server

Use ' dispatch help [command] ' For more information about a command.

Vi. Summary

The full source of the above example can be found in the author's GitHub repository.

With regard to the Golang profile, I personally used the TOML level because there is no need for too complex configurations, no environment variables or command line override defaults or profile data. However, from the author's example, we can see that multiconfig, Viper is indeed powerful, and subsequent implementations of complex Golang applications will be considered for real applications.

Bigwhite. All rights reserved.