Some ideas on improving log4go

This is a creation in Article, where the information may have evolved or changed.

After the release of the Log4go 4.0.2 version (https://github.com/ccpaging/log4go/tree/4.0.2),
Look at the other Go language log file design. Found a good article:

Comparison and analysis of Log4go and Logrus
Https://www.doraemonext.com/archives/783.html

The clues, found a nest about the design of the log. The links are as follows (including old links):

1. https://github.com/alecthomas/log4go/
   This is the "originator" of the Log4go project.
2. Https://github.com/ngmoco/timber
   Realize the structure, write file buffer, thermal configuration and so on. To reconstruct the log4go beyond recognition.
3. Https://github.com/siddontang/go/tree/master/log
4. Https://github.com/sirupsen/logrus
5. Https://github.com/YoungPioneers/blog4go
6. Https://github.com/YoungPioneers/blog4go-benchmark benchmark comparison of various go logs
7. Https://github.com/cihub/seelog

Write Log asynchronously

One of the features of Log4go is asynchronous writing. Formatting logging, writing files, dumping logs, and so on, consumes CPU time and may block the main thread because of error handling.
However, the log system is only an auxiliary function, so to ensure the efficient operation of the main thread is the first to achieve the design requirements. Asynchronous write is one of the possible scenarios.

Self-Expanding log interface

In fact, Log4go is a support for similar Logrus extension features.

It's just a matter of how the design of the color text term log is handled ... Because this feature uses a third-party package. In the log4go, it increased its dependence. But this is really a feature I particularly like.

Instead, make the color text term log an extended log interface. Do what you say ...

Let's figure out the extension interfaces available in Log4go:

type  logwriter  Interface  {logwrite (rec *logrecord) //this should clean up anything lingering about the Lo  Gwriter, as it is called before  //the LogWriter is removed.    Logwrite should not being called after Close.  Close ()}type  Filter struct  {level level rec ch  An  *logrecord //write queue  closed bool  // True if Socket is closed at API level  logwriter}type  Logger map[string ]*filterfunc  (log Logger) addfilter (name  String , lvl level, writer LogWriter) Logger {Log[name] = Newfilter (LVL, writer) return log}  

Extensions just do:

1. NewXXXLogWrite, initialize the resources to be used by the extension.
2. LogWrite(rec *LogRecord), Output log records
3. To Close() close or release resources in
4. Call in the application to AddFilter add a new log extension to the LOG4GO log structure

It's done.

Where the ADD filter name is the index keyword for Logger map, Log4go uses:

"stdout", "File", "Syslog"

If the new Filter keyword already exists, the Log4go (later version of 4.0.2) will automatically close the original, and then add the new. The code is as follows:

funcstring, lvl Level, writer LogWriter) Logger {    if filt, isExist := log[name]; isExist {        filt.Close()        delete(log, name)    }    log[name] = NewFilter(lvl, writer)    return log}

With the extension interface, Log4go's logging can take any encapsulation format you want, such as XML and JSON, which is already implemented.
You can also extend the CSV later (import the log file into Excel) or the JSON encapsulated message.

The extensible Log interface includes:

• Send error messages as a mail

• Make TCP/UDP server and let client pull the messages

• WebSocket

• Nanomsg pub/sub

• Store log messages in MySQL

Self-extending Log Configuration Interface

Log4go 4.0.2 supports XML and JSON configurations. There are three ways to configure log files:

1. Configuring in the Application
2. Separate configuration files
3. stored in the main program configuration file

As an auxiliary function, the log system is often confronted with a third case. The configuration file has a variety of formats. For example:

Windows INI, Linux Config, JSON, XML ...

Depressed. Log4go should not support all the configuration file formats, but rather provide interfaces that allow users to expand themselves according to the design needs of their main program .

Perhaps the XML and JSON configuration file support should be implemented in a way that extends the configuration file interface, rather than bundled with Log4go's main program.

Write buffering for file logs

A two-layer buffered write file has been tested.

The first layer is formatted log records, a separate go routine, another write file, Edge format recording edge write file, consumption decreased by 40%.

The second floor is with Bufio. To achieve a certain number of buffers such as 4k, 8k, write a file at once. consumption decreased by 80%.

Determines when the system is free by judging the length of records in the channel. When the length is 0 o'clock, no new log records are made, and flush () is done once.
This scheme is simple.

Plus rotate optimization, the efficiency increased 5 times times.

BenchmarkFileLog-4                        200000             10675 ns/opBenchmarkFileNotLogged-4                20000000               106 ns/opBenchmarkFileUtilLog-4                    200000             10660 ns/opBenchmarkFileUtilNotLog-4                5000000               239 ns/opBenchmarkCacheFileLog-4                  1000000              2191 ns/opBenchmarkCacheFileNotLogged-4           20000000               106 ns/opBenchmarkCacheFileUtilLog-4               500000              3680 ns/opBenchmarkCacheFileUtilNotLog-4           5000000               240 ns/op

A tentative idea of Rotate improvement

Log4go comes with rotate.

The Linux system was originally logrotate and executed with Cron. Very good design.
Simply put, write log file to write log file, do not do any dump judgment. The programmer can, according to the actual operation of the system,
The time interval at which to set the dump yourself. When dumping:

• Locking Make Log4go temporarily stop writing logs, which may be one of the reasons Log4go not using Logrotate on Linux systems.

• The current log file is processed.

• Unlock. Resume Log4go as soon as possible and continue writing logs to the current log file.

• Open go routine to process the history log file.

All right. I think so much for the time being. A lot of interesting work is going on ...

Thanks again for the Good article of doraemonext@gmail.com children's shoes: The comparison and analysis of Log4go and Logrus

Please pay attention to:

Https://github.com/ccpaging/log4go