How do I use the Golang log to monitor your application?

This is a creation in Article, where the information may have evolved or changed.

Guide

Are you tired of the hard-to-deploy, always-on-build solutions written in complex languages? Golang is a great way to solve these problems, as fast as the C language and as simple as Python.

But how do you use the Golang log to monitor your application? Golang no exception, only error. So your first impression might be that developing a Golang logging strategy is not an easy task. Not supporting exceptions is actually not a problem, and exceptions have been lost in many programming languages: they are so abused that their role is ignored.

Before going any further, we'll start by introducing the basics of the Golang log and discussing the Golang log standards, metadata meanings, and the impact of minimizing Golang logs on performance. Logs allow you to track user activity in your app, quickly identify components that are failing in your project, and monitor overall performance and user experience.

I. Golang Log Basics

Using the Golang "log" library

Golang provides you with a native log library called "Log". Its logger is perfect for tracking simple activities, such as adding a timestamp before the error message by using the available options.

Here is a simple example of how to log error logs in a Golang:

Package Mainimport (    "Log"    "errors" "    fmt"    ) Func main () {/   * define local variable */  ...   /* The Division function, divided by 0, will return an error *   /ret,err = div (A, b) if err! = Nil {log. Fatal (Err)    }    fmt. PRINTLN (ret)}

If you try to divide by 0, you will get a result similar to the following:

To quickly test a golang function, you can use Go playground.

To ensure that your logs are always easy to access, we recommend that you write them to a file:

Package Mainimport (        "Log"        "OS") Func main () {        //create file F with required read        and Write permissions, err: = OS. OpenFile ("filename", os. O_wronly|os. O_create|os. O_append, 0644)        if err! = Nil {                log. Fatal (Err)        }           //After completion delay closing, not habit!        Defer f.close ()        //set log output to F        log. Setoutput (f)        //test Case        log. Println ("Check to make sure it works")}

You can find the complete guide to the Golang log and a complete list of the functions available in the "Log" library.

Now you can record their errors and the root cause.

In addition, the log can help you to stitch the active stream together, find the error context to be repaired, or investigate how individual requests in your system affect other application tiers and APIs.

To get a better log effect, you first need to enrich your Golang log with as many contexts as possible in your project, and standardize the format you use. This is the limit that can be reached by the Golang native library. The most widely used libraries are glog and Logrus. It must be admitted that there are many good libraries to use. If you're already using a library that supports JSON format, you don't need to switch to another library, and we'll explain later.

II. Golang for you log Unified format

1) Structure Advantages of JSON format

Structuring your Golang logs in a project or in multiple microservices can be the hardest thing to do, but it's easy to do once you're done. Structured your logs can make machine readable (refer to our best practices blog for collecting logs). Flexibility and hierarchy are at the heart of the JSON format, so information can be easily parsed and processed by humans and machines.

Here is an example of using Logrus/logmatic.io to log logs in JSON format:

Package Mainimport (  log "Github.com/sirupsen/logrus"  "Github.com/logmatic/logmatic-go") func main () {    / /use Jsonformatter    log. Setformatter (&logmatic. jsonformatter{})        //Use Logrus to log event log as usual    . Withfields (log. fields{"string": "foo", "int": 1, "float": 1.1}). Info ("My first SSL event from Golang")}

Results are output:

{       "date": "2016-05-09t10:56:00+02:00",    "float": 1.1,    "int": 1,    "level": "Info",    "message": "My First SSL event from Golang ",    " String ":" foo "}

2) Standardized Golang log

It is shameful that the same error appears in different parts of your code, but is recorded in different forms. The following is an example of a variable error that could not determine the load status of a Web page. A developer log format is:

Message: ' Unknown error:cannot determine loading status from unknown error:missing or invalid arg value client ' </span >

Another person's format is:

Unknown error:cannot determine loading status-invalid client</span>

A good solution to enforcing log normalization is to create an interface between your code and the log library. This standardized interface will include all predefined log messages that you want to add to your log for possible behavior. Doing so prevents the custom log information from appearing in the standard format that you want. This also facilitates log investigation.

Because the log formats are processed uniformly, it is easier to keep them updated. If a new type of error occurs, it only needs to be added to an interface so that each member will use the exact same information.

A simple example that is most commonly used is to add the logger name and ID before the Golang log message. Your code will then send "events" to your standardized interface, which will continue to translate them into Golang log messages.

In the main section, we'll define all the messages here. The Event structure is simple. To retrieve them when all information is recorded,//We maintain a idvar (    invalidargmessage = event{1, "Invalid arg:%s"}    invalidargvaluemessage = Event{2, "Invalid arg value:%s =%v"}    missingargmessage = event{3, "Missing arg:%s"})//All log event Func that can be used in our application (l *logger) Invalidarg (name string) {    L.entry.errorf (invalidargmessage.tostring (), name)}func (l *logger) Invalidargvalue ( Name string, value interface{}) {    L.entry.withfield ("Arg." + Name, value). Errorf (Invalidargvaluemessage.tostring (), name, value)}func (l *logger) missingarg (name string) {    L.entry.errorf ( Missingargmessage.tostring (), name)}

So if we use an invalid parameter value from the previous example, we get similar log information:

Time= "2017-02-24t23:12:31+01:00" Level=error msg= "loadpagelogger00003-missing arg:client-cannot determine loading St ATUs "Arg.client=<nil> Logger.name=loadpagelogger

The JSON format is as follows:

{"arg.client": null, "Level": "Error", "Logger.name": "Loadpagelogger", "MSG": "Loadpagelogger00003-missing arg:client -Cannot determine loading status "," Time ":" 2017-02-24t23:14:28+01:00 "}

Iii. Golang The power of the log context

Now that the Golang log has been recorded in a specific structure and standard format, time determines what context and related information needs to be added. In order to be able to extract information from your logs, such as tracking a user activity or workflow, the context and order of the metadata is important.

For example, in the Logrus library, you can add hostname, appname, and session parameters using the JSON format as follows:

For metadata, the usual practice is to reuse the fields in the log statements by reusing them.  Contextualizedlog: = log. Withfields (log. fields{    "hostname": "Staging-1", "    appname": "Foo-app",    "session": "1ce3f6v"  }) Contextualizedlog.info ("Simple event with global metadata")

Metadata can be considered a JavaScript fragment. To better illustrate how important they are, let's look at the use of Meta data in several Golang microservices. You will clearly see how the user is tracked in your application. This is because you need not only to know that an error has occurred, but also to know which instance and what mode caused the error. Let's say we have two micro services that are called sequentially. The context information is saved in the header (header) for transmission:

Func HelloMicroService1 (w http. Responsewriter, R *http. Request) {client: = &http. client{}//The service is responsible for receiving all incoming user requests//We will check whether it is a new session or a session that has another call session: = R.header.get ("X-session") if (session = = "") { Session = Generatesessionid ()//record log for new session}//each request track Id is unique, so we generate a track for each session: = Generatetrackid ()//Call your second micro-service, Add Session/trackreqservice2, _: = http. Newrequest ("GET", "http://localhost:8082/", nil) reqService2.Header.Add ("X-session", session) REQSERVICE2.HEADER.ADD ("X-track", track) ResService2, _: = client. Do (REQSERVICE2) ....

When a second service is called:

Func HelloMicroService2 (w http. Responsewriter, R *http. Request) {//similar to the previous microservices, we check the session and generate a new tracksession: = R.header.get ("X-session") Track: = Generatetrackid ()//This time, We check whether a track id,//has been set in the request if yes, it becomes the parent trackparent: = R.header.get ("X-track") if (session = = "") {W.header (). Set ("X-parent", parent)}//adds meta information to the response W. Header (). Set ("X-session", session) W.header (). Set ("X-track", track) if (parent = = "") {W.header (). Set ("X-parent", track)}//fill response W. Writeheader (http. Statusok) io. WriteString (W, FMT. Sprintf (Aresponsemessage, 2, session, track, parent))}

Now that the second microservices already have context and information related to the initial query, a JSON-formatted log message looks similar to the following.

In the first micro-service:

{"AppName": "Go-logging", "Level": "Debug", "MSG": "Hello from Ms 1", "Session": "EUBRVFDW", "Time": "2017-03-02t15:29:26+ 01:00 "," track ":" UZWHRIHF "}

In the second micro-service:

{"AppName": "Go-logging", "Level": "Debug", "MSG": "Hello from Ms 2", "Parent": "UZWHRIHF", "Session": "EUBRVFDW", "Time": " 2017-03-02t15:29:26+01:00 "," track ":" Dprhbmue "}

If an error occurs in the second microservices, thanks to the context information saved in the Golang log, we can now determine how it was called and what pattern caused the error.

If you want to dig deeper into Golang's tracking capabilities, there are also some libraries that provide tracking capabilities, such as opentracing. This library provides an easy way to add tracking implementations in a complex or simple architecture. It allows you to track users ' queries through different steps, like this:

Iv. Golang Log impact on performance

1) Do not use the log in Goroutine

It looks tempting to create a new logger in each goroutine. But it's better not to do so. Goroutine is a lightweight thread manager that is used to complete a "simple" task. Therefore it should not be responsible for logs. It can cause concurrency problems because it is used in each goroutine

Log. New ()

Will repeat the interface, and all the logger will attempt to access the same IO concurrently. Writer.

To limit the impact on performance and avoid concurrent calls to IO. Writer, libraries typically use a specific goroutine for log output.

2) using the asynchronous library

Although there are many Golang log libraries available, it is important to note that most of them are synchronous (in fact, pseudo-async). The reason is probably that none of them so far has affected performance because of the log.

But as Kjell Hedström in his experiment, creating thousands of logs with multiple threads, even in the worst case scenario, has a 40% performance boost for asynchronous Golang logs. So logs are expensive and can have an impact on the performance of your application. If you do not need to handle a large number of logs, it may be sufficient to use a pseudo-asynchronous Golang log Library. But if you need to deal with a large number of logs, or focus on performance, Kjell Hedström's asynchronous solution is interesting (although in fact you may need to develop further because it includes only the smallest functional requirements).

3) Manage Golang logs using severity level

Some log libraries allow you to enable or disable specific logger, which may come in handy. For example, in a production environment, you may not need a certain level of logging. Here is an example of how to deactivate the logger in the Glog library, where the logger is defined as a Boolean value:

Type Log Boolfunc (l log) Println (args ... interface{}) {    fmt. Println (args ...)} var debug Log = Falseif Debug {    Debug. Println ("Debugging")}

You can then define these Boolean parameters in the configuration file to enable or deactivate the logger.

Without a good Golang log policy, the Golang log can be expensive. Developers should resist the temptation to record almost everything-though it's very interesting! If the purpose of the log is to obtain as much information as possible, in order to avoid the white noise of logs that contain useless elements, the log must be used correctly.

V. Centralize Golang logs

If your application is deployed on more than one server, you can avoid the hassle of investigating a phenomenon that requires connecting to each server. The log set is really useful.

Using the Log Boxing tool, such as Rsyslog in Windows Nxlog,linux (installed by default), Logstash, and Fluentd, is the best way to implement it. The only purpose of the Log Boxing tool is to send logs, so they are able to handle connection failures and other problems that you are likely to encounter.

There is even a Golang syslog package that helps you send the Golang log to the syslog daemon.

Hope you enjoy your Golang log tour

It is important to consider your Golang log strategy at the outset of your project. If you get all the context anywhere in your code, it's easy to keep track of the user. It is already painful to read from different services without standardized logs. Plan to extend the same user or request ID in multiple microservices at the outset, allowing you to easily filter information and track activity in your system.

Whether you are framing a large Golang project or a few microservices can also affect your logging strategy. The main components of a large project should have a specific Golang logger named after their function. This allows you to immediately determine which part of your code the log is from. However, for microservices or small golang projects, only a few core components require their own logger. In each case, however, the number of logger should remain below the number of core functions.

You can now use Golang log quantization to determine your performance or user satisfaction!

via:https://logmatic.io/blog/our-guide-to-a-golang-logs-world/

Author: Nils Translator: Ictlyh proofreading: Wxy

This article was compiled by LCTT original, Linux China honors launched

Originally from: https://www.linuxprobe.com/go-language-log.html