Log practices that every Python programmer needs to know

In real life, logging is important. Bank transfer will have a record of the transfer, the aircraft during the flight, there will be a black box (flight data logger) to record everything during the flight. If there are any problems, people can use log data to figure out what exactly happened.

Logging is equally important for system development, commissioning, and operation. Without logging, you can hardly figure out what's going on when the program crashes. For example, logging is necessary when you are writing a server program. The log files for the ezcomet.com server are shown below.

After the service crashed, if there were no logs, I could hardly know what was wrong. Logging is important not only for servers, but also for desktop graphics applications. For example, when your client's PC program crashes, you can ask them to send you a log file so you can find out where the problem is. Believe me, in a different PC environment, you will never know what the strange problem is. I have received such an error log.

Python

1
2
3
4
5
6
7
8
9
10
11
12
13
14
2011-08-22 17:52:54,828-root-error-[Errno 10104] getaddrinfo failed
Traceback (most recent):
File "<string>", line 124, in main
File "<string>", line A, in __init__
File "H:workspaceprojectbuildpyi.win32mrdjoutpyz1.pyz/wx._core", line 7978, in __init__
File "H:workspaceprojectbuildpyi.win32mrdjoutpyz1.pyz/wx._core", line 7552, in _bootstrapapp
File "<string>", line +, in OnInit
File "H:workspaceprojectbuildpyi.win32mrdjoutpyz1.pyz/twisted.internet.wxreactor", line 175, in install
File "H:workspaceprojectbuildpyi.win32mrdjoutpyz1.pyz/twisted.internet._threadedselect", line 106, in __init__
File "H:workspaceprojectbuildpyi.win32mrdjoutpyz1.pyz/twisted.internet.base", line 488, in __init__
File "H:workspaceprojectbuildpyi.win32mrdjoutpyz1.pyz/twisted.internet.posixbase", line 266, in InstallWaker
File "H:workspaceprojectbuildpyi.win32mrdjoutpyz1.pyz/twisted.internet.posixbase", line-up, in __init__
File "H:workspaceprojectbuildpyi.win32mrdjoutpyz1.pyz/socket", line 224, in meth
Gaierror: [Errno 10104] getaddrinfo failed
I finally found out that this client's PC was infected by a virus that caused the call to GetHostName function to fail. Look, if there's no log, you can find out how you might know that.

PrintOut is not a good idea.

Although logging is important, not all developers are able to use it correctly. I've seen some developers log on like this, insert print statements during development, and then remove these statements after development. Just like this:

Python

1
2
3
4
5
6
print ' Start reading database '
Records = Model.read_recrods ()
print ' # Records ', records
print ' Updating record ... '
Model.update_records (Records)
print ' Done '
This is useful for simple scripting programs, but if it's a complex system, you might as well not use it this way. First of all, you have no way to leave only extremely important messages in the log file. You'll see a lot of message logs. But you can't find any useful information. There's no way you can control the code except to remove the output statement, but it's very likely that you forgot to remove the useless output. Furthermore, all the information from the print output is in the standard output, which will seriously affect your viewing of other output data from the standard output. Of course, you can also output the message to stderr, but it's not good to use print as a way to log records.

Use Python's standard log module

So, what is the correct way to log logs? In fact, it's very simple, using Python's standard log module. Thanks to the Python community, the logs are made into a standard module. It is very easy to use and very flexible. You can use the log system like this:

Python

1
2
3
4
5
6
7
8
9
10
11
12
13
Import logging
Logging.basicconfig (Level=logging.info)
Logger = Logging.getlogger (__name__)

Logger.info (' Start reading database ')
# Read Database here

Records = {' John ':, ' Tom ': 66}
Logger.debug (' Records:%s ', Records)
Logger.info (' Updating records ... ')
# Update records here

Logger.info (' Finish Updating Records ')
When you run it, you can see:

Python

1
2
3
Info:__main__:start Reading Database
Info:__main__:updating Records ...
Info:__main__:finish Updating records
You might ask how this is different from using print. It has the following advantages:

You can control the level of messages and filter out messages that are not important.
You can decide where to export and how to output it.
There are many important levels to choose from, Debug, info, warning, error, and critical. By giving logger or handler different levels, you can output only error messages to a particular record file, or only debug information when debugging. Let's change the logger level to DEBUG and look at the output:

Python

1
Logging.basicconfig (level=logging. DEBUG)
The output becomes:

Python

1
2
3
4
Info:__main__:start Reading Database
Debug:__main__:records: {' john ':, ' Tom ': 66}
Info:__main__:updating Records ...
Info:__main__:finish Updating records
As you can see, when we change the logger level to debug, the debug record appears in the output. You can also choose how to handle these messages. For example, you can use Filehandler to write records into a file:

Python

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
Import logging

Logger = Logging.getlogger (__name__)
Logger.setlevel (Logging.info)

# Create a file handler

Handler = logging. Filehandler (' Hello.log ')
Handler.setlevel (Logging.info)

# Create a logging format

Formatter = logging. Formatter ('% (asctime) s-% (name) s-% (levelname) s-% (message) s ')
Handler.setformatter (Formatter)

# Add the handlers to the logger

Logger.addhandler (Handler)

Logger.info (' Hello baby ')
The standard library module provides a number of handler that you can send records to a mailbox or even to a remote server. You can also implement your own record handler. Details of the implementation will not be described here, you can refer to the official documentation: BASCI turial, Advanced Tutorial and Logging Cookbook.

Output log records at the appropriate level

With the flexible logging module, you can output logging to any location and configure them at the appropriate level. So you might ask, what is the right level? I will share some of my experience here.

In most cases, you don't want to read too much detail in the log. Therefore, only you will use the debug level during the debugging process. I only use debug to get detailed debugging information, especially when the amount of data is large or very high, such as the intermediate state of each loop within the algorithm.

Python

1
2
3
4
5
def complex_algorithm (items):
For I, item in enumerate (items):
# do some complex algorithm computation

Logger.debug ('%s iteration, item=%s ', I, item)
I use the INFO level in everyday transactions such as request processing or server status changes.

Python

1
2
3
4
5
6
7
8
9
10
11
def handle_request (Request):
Logger.info (' Handling request%s ', request)
# Handle Request Here

result = ' result '
Logger.info (' Return Result:%s ', result)

Def start_service ():
Logger.info (' Starting service at Port%s ... ', port)
Service.start ()
Logger.info (' Service is started ')
I use WARNING when there are important events, but not errors. For example, when a user logs in a bad password, or when the connection slows down.

Python

1
2
3
4
5
Def authenticate (user_name, password, ip_address):
If user_name! = user_name and password! = password:
Logger.warn (' Login attempt to%s from IP%s ', user_name, ip_address)
Return False
# do authentication here
Error levels are definitely used when errors occur. such as throwing exceptions, IO operations failed, or connection problems.

Python

1
2
3
4
5
6
def get_user_by_id (user_id):
user = Db.read_user (user_id)
If User is None:
Logger.error (' cannot find user with user_id=%s ', user_id)
return user
return user
I seldom use CRITICAL. When something really bad happens, you can use that level to record it. For example, memory exhaustion, disk full or nuclear crisis (Hope never happens: S).

Use __name__ as the name of the logger

Although it is not mandatory to set the name of logger to __name__, doing so will bring us many benefits. In Python, the name of the variable __name__ is the name of the current module. For example, calling Logger.getlogger (__name__) in the module "Foo.bar.my_module" is equivalent to calling Logger.getlogger ("Foo.bar.my_module"). When you need to configure logger, you can configure it to "Foo" so that all modules in the package Foo will use the same configuration. When you read a log file, you can see exactly which module the message came from.

Catch the exception and use Traceback to record it

It's a good habit to write down when something goes wrong, but if you don't have a traceback, it doesn't work at all. You should catch exceptions and record them with Traceback. For example, the following:

Python

1
2
3
4
5
6
Try
Open ('/path/to/does/not/exist ', ' RB ')
Except (Systemexit, Keyboardinterrupt):
Raise
Except Exception, E:
Logger.error (' Failed to open file ', exc_info=true)
Using the parameter exc_info=true to call the Logger method, the Traceback is output to logger. You can see the following results:

Python

1
2
3
4
5
error:__main__:failed to open File
Traceback (most recent):
File "example.py", line 6, <module>
Open ('/path/to/does/not/exist ', ' RB ')
IOError: [Errno 2] No such file or directory: '/path/to/does/not/exist '
You can also call Logger.exception (MSG, _args), which is equivalent to Logger.error (msg, exc_info=true, _args).

Never get Logger at the module level unless disable_existing_loggers is set to False

You can see many examples of getting logger at the module level (I've used a lot in this article, but just to make the example shorter). They look harmless, but in fact, there's a trap here – if you use Logger,python like this in a module, all the Logger created before the configuration is read from the file are preserved.

my_module.py

Python

1
2
3
4
5
6
7
8
9
10
Import logging

Logger = Logging.getlogger (__name__)

def foo ():
Logger.info (' Hi, foo ')

Class Bar (object):
def bar (self):
Logger.info (' Hi, Bar ')
main.py

Python

1
2
3
4
5
6
7
8
9
10
Import logging

Logger = Logging.getlogger (__name__)

def foo ():
Logger.info (' Hi, foo ')

Class Bar (object):
def bar (self):
Logger.info (' Hi, Bar ')
Logging.ini

Python

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21st
22
[Loggers]
Keys=root

[Handlers]
Keys=consolehandler

[Formatters]
Keys=simpleformatter

[Logger_root]
Level=debug
Handlers=consolehandler

[Handler_consolehandler]
Class=streamhandler
Level=debug
Formatter=simpleformatter
Args= (Sys.stdout,)

[Formatter_simpleformatter]
format=% (asctime) s-% (name) s-% (levelname) s-% (message) s
datefmt=
I should have seen the record in the log, but you didn't see anything. Why is it? This is because you created the logger at the module level, and then you imported the module before loading the log configuration file. Logging.fileconfig and Logging.dictconfig By default will invalidate existing logger. Therefore, these configuration information will not be applied to your Logger. You'd better only get it when you need logger. The cost of creating or acquiring logger is very low anyway. You can write your code like this:

Python

1
2
3
4
5
6
7
8
9
10
11
12
Import logging

def foo ():
Logger = Logging.getlogger (__name__)
Logger.info (' Hi, foo ')

Class Bar (object):
def __init__ (self, Logger=none):
Self.logger = Logger or Logging.getlogger (__name__)

def bar (self):
Self.logger.info (' Hi, Bar ')
This way, logger will not be created until you load the configuration. This allows configuration information to be applied as expected.

In the later version of python2.7, FILECONFG and Dictconfig both added the "disable_existing_loggers" parameter and set it to False, the problem mentioned above can be resolved. For example:

Python

1
2
3
4
5
6
7
8
9
Ten
One


20


(
)
,
,
,
,
,
,
,
,
, and
, import Logging
Import logging.config

Logger = Logging.getlogger (__name__)

# Load config from file

# l Ogging.config.fileConfig (' Logging.ini ', disable_existing_loggers=false)

# or, for Dictconfig

Logging.config.dictConfig ({
' version ': 1,
' disable_existing_loggers ': False, # This fixes the problem

' F Ormatters ': {
' standard ': {
' format ': '% (asctime) s [% (levelname) s]% (name) s:% (message) s '
},
},
' hand Lers ': {
' default ': {
' level ': ' INFO ',
' class ': ' Logging. Streamhandler ',
},
},
' loggers ': {
': {
' handlers ': [' Default '],
' level ': ' INFO ',
' Propag Ate ': True
}
}
}

Logger.info (' It works! ')
Configure with JSON or YAML records

Although you can configure your log system in Python code, this is not flexible enough. The best approach is to configure it with a configuration file. In Python2.7 and later versions, you can load the logging configuration from the dictionary. This also means that you can load the configuration of the log from a JSON or YAML file. Although you can also use the original. ini file to configure, but it is both difficult to read and difficult to write. Let me show you a sample configuration using JSON and YAML files:

Logging.json

Python

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21st
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
{
"Version": 1,
"Disable_existing_loggers": false,
"Formatters": {
"Simple": {
"Format": "% (asctime) s-% (name) s-% (levelname) s-% (message) S"
}
},

"Handlers": {
"Console": {
"Class": "Logging." Streamhandler ",
"Level": "DEBUG",
"Formatter": "Simple",
"Stream": "Ext://sys.stdout"
},

"Info_file_handler": {
"Class": "Logging.handlers.RotatingFileHandler",
"Level": "INFO",
"Formatter": "Simple",
"FileName": "Info.log",
"MaxBytes": 10485760,
"Backupcount": 20,
"Encoding": "UTF8"
},

"Error_file_handler": {
"Class": "Logging.handlers.RotatingFileHandler",
"Level": "ERROR",
"Formatter": "Simple",
"FileName": "Errors.log",
"MaxBytes": 10485760,
"Backupcount": 20,
"Encoding": "UTF8"
}
},

"Loggers": {
"My_module": {
"Level": "ERROR",
"Handlers": ["console"],
"Propagate": "No"
}
},

"Root": {
"Level": "INFO",
"Handlers": ["Console", "Info_file_handler", "Error_file_handler"]
}
}
Logging.yaml

Python

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21st
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
---

Version:1

Disable_existing_loggers:false

Formatters:

Simple

Format: "% (asctime) s-% (name) s-% (levelname) s-% (message) S"

Handlers:

Console

Class:logging. Streamhandler

Level:debug

Formatter:simple

Stream:ext://sys.stdout

Info_file_handler:

Class:logging.handlers.RotatingFileHandler

Level:info

Formatter:simple

Filename:info.log

maxbytes:10485760 # 10MB

Backupcount:20

Encoding:utf8

Error_file_handler:

Class:logging.handlers.RotatingFileHandler

Level:error

Formatter:simple

Filename:errors.log

maxbytes:10485760 # 10MB

Backupcount:20

Encoding:utf8

Loggers:

My_module:

Level:error

Handlers: [Console]

Propagate:no

Root

Level:info

Handlers: [Console, Info_file_handler, Error_file_handler]

...
Next, you'll show how to read the configuration information from the JSON file into the log:

Python

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21st
Import JSON
Import Logging.config

Def setup_logging (
Default_path= ' Logging.json ',
Default_level=logging.info,
env_key= ' Log_cfg '
):
"" "Setup Logging Configuration

"""
Path = Default_path
Value = Os.getenv (Env_key, None)
If value:
Path = value
If Os.path.exists (path):
With open (PATH, ' RT ') as F:
Config = json.load (f)
Logging.config.dictConfig (config)
Else
Logging.basicconfig (Level=default_level)
One advantage of using JSON is that JSON is a standard library, and you don't need to install it any additional. But personally, I like YAML a little bit. It is easier to read or write. You can also use the following method to load a YAML configuration file:

Python

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21st
22
23
Import OS
Import Logging.config

Import Yaml

Def setup_logging (
Default_path= ' Logging.yaml ',
Default_level=logging.info,
env_key= ' Log_cfg '
):
"" "Setup Logging Configuration

"""
Path = Default_path
Value = Os.getenv (Env_key, None)
If value:
Path = value
If Os.path.exists (path):
With open (PATH, ' RT ') as F:
Config = yaml.load (F.read ())
Logging.config.dictConfig (config)
Else
Lo
Next, you can invoke setup_logging to start logging when you run the program. It reads Logging.json or logging.yaml files by default. You can also set the environment variable LOG_CCFG to load the log configuration from the specified path. For example:

Python

1
Log_cfg=my_logging.json python my_server.py
If you like YAML:

Python

1
Log_cfg=my_logging.yaml python my_server.py
Using the Rotate file handle

If you write logs with Filehandler, the size of the file will increase over time. Eventually it will fill up all of your disk space one day. To prevent this from happening, you can use Rotatingfilehandler instead of Filehandler in your build environment.

If you have multiple servers, you can enable a dedicated log server

When you have multiple servers and different log files, you can create a centralized logging system to collect important (mostly warning or error messages) information. Then by monitoring these log messages, you can easily discover the problems in the system.

Summarize

Python's log library is so well designed, it's comforting, I think it's the best part of the standard library and you have to choose it. It's very flexible and you can use your own handler or filter. There are already a lot of third-party handler, such as the ZeroMQ log handle provided by PYZMQ, which allows you to send log messages through ZMQ sockets. This article will be useful if you do not yet know how to use the journaling system correctly. With good logging practices, you'll be able to find problems in your system very easily. This is very well worth investing in. :)

Log practices that every Python programmer needs to know