Let's talk about Python logging and Pythonlogging.

Let's talk about Python logging and Pythonlogging.

Recently, it was necessary to change the log output from the previous string to the json format. After reading other people's examples, it was still a bit confusing. Simply put the entire logging aside and make a summary.

First look at log

In the program, you can use log to write code. You know that log has a level, DEBUG, INFO ,... and so on. It also records the time and location where logs occur. In Python, most of them are the packages in the standard library of logging. what happened when a log was played? How to output logs of different levels to different files and output logs in the console .......

Simplest usage

Import logging
Logging. basicConfig (format = '% (levelname) s: % (message) s', level = logging. DEBUG)
Logging. debug ('this message shocould go to the log file ')
Logging.info ('so shoshould this ')
Logging. warning ('And this, too ')
1. The first line imports package 2, and the second line uses basicConfig to limit the output format and output level 3. Three log levels are output respectively.

Logging Levels

There are several levels. Each level corresponds to an integer of the Int type, and each level corresponds to a method, so that the output content has different levels.

Logger process,

The whole process is still not very detailed. paste a picture. It's still too early to tell what happened in the real process. Let's put it first. It will be easier to understand later. loger flow

Read code

Code structure

Logging has three files in the source code. The structure is as follows:

── Config. py
── Handlers. py
── _ Init _. py
_ Int. py implements basic functions, and the main logic is handlers in this file. py is very convenient for some Handlers. config. py is the method for processing the configuration.

Objects

LogRecord Objects

Every time a log is instantiated, a Record object has many attributes. At last, the format of the LogRecord is output, and the formatted log is basically the attribute of the object.

class LogRecord(object):  def __init__(self, name, level, pathname, lineno,         msg, args, exc_info, func=None):    ct = time.time()    self.name = name    self.msg = msg    if (args and len(args) == 1 and isinstance(args[0], collections.Mapping)      and args[0]):      args = args[0]    self.args = args    self.levelname = getLevelName(level)    self.levelno = level    self.pathname = pathname    try:      self.filename = os.path.basename(pathname)      self.module = os.path.splitext(self.filename)[0]    except (TypeError, ValueError, AttributeError):      self.filename = pathname      self.module = "Unknown module"    self.exc_info = exc_info    self.exc_text = None   # used to cache the traceback text    self.lineno = lineno    self.funcName = func    self.created = ct    self.msecs = (ct - long(ct)) * 1000    self.relativeCreated = (self.created - _startTime) * 1000    if logThreads and thread:      self.thread = thread.get_ident()      self.threadName = threading.current_thread().name    else:      self.thread = None      self.threadName = None    if not logMultiprocessing:      self.processName = None    else:      self.processName = 'MainProcess'      mp = sys.modules.get('multiprocessing')      if mp is not None:        try:          self.processName = mp.current_process().name        except StandardError:          pass    if logProcesses and hasattr(os, 'getpid'):      self.process = os.getpid()    else:      self.process = None  def __str__(self):    return '<LogRecord: %s, %s, %s, %s, "%s">'%(self.name, self.levelno,      self.pathname, self.lineno, self.msg)  def getMessage(self):     pass

When I look at the code, I find that this class has nothing to do. It is just a model and there is a method to get msg.

Formatter Objects

Formatter is an object specially formatted for Record. It has a format method, so we can implement this method to achieve different outputs, my requirement is that the key to creating a log in json format is to write a Formatter.

class Formatter(object):  converter = time.localtime  def __init__(self, fmt=None, datefmt=None):    if fmt:      self._fmt = fmt    else:      self._fmt = "%(message)s"    self.datefmt = datefmt  def formatTime(self, record, datefmt=None):    pass  def formatException(self, ei):    pass  def usesTime(self):    return self._fmt.find("%(asctime)") >= 0  def format(self, record):    pass

Delete the implementation details in the source code. The main method in this class is the format method, which is the most basic Formater by default. There is also a special method for formatting the exception and time. The specific method is clearly indicated by the method name. You can understand how each method is implemented at a glance. Fmt is formatted. How to specify it is shown in the most basic usage example. datefmt specifies the time format.

Filter Objects

This class is the base class of Logger and Handler. It mainly includes a Filter method and a filters attribute.

Handler Objects

There are actually a lot of classes called Handler. We can also see in SocketServer that the specific functions are in Handler. here, all formatters are combined and the output direction of the control log is inherited from the Filter.

 def __init__(self, level=NOTSET):    Filterer.__init__(self)    self._name = None    self.level = _checkLevel(level)    self.formatter = None    _addHandlerRef(self)    self.createLock()

In the init method, Handler also has an attribute. It compares its own attribute with the level of LogRecord to determine whether to process the LogRecord. Each Handler has a Formatter attribute, which is actually the Formatter described above. Handler is used to control the LogRecord and Formatter. It can also control the output mode. There will be StreamHandler and FileHandler in the future. By name, you can understand what you can do. This is the wisdom of programming naming.

Logger Objects

This class is usually obtained through getLogger () or getLogger (name) and will not be new directly. it has methods such as info (msg, * args, kwargs), warn (msg, args, * kwargs,

  def __init__(self, name, level=NOTSET):    Filterer.__init__(self)    self.name = name    self.level = _checkLevel(level)    self.parent = Noneou    self.handlers = []    self.disabled = 0

The handlers attribute can be seen from the init method. This is a list. Each LogRecord can be output to different places in different formats through different Handlers. Each Logger can add various Handler via the addHandler (hdlr) method. If you know this, you can customize it as needed. The following is the Formater in the json format that I implemented. It supports console color changes, of course, the premise is that your control terminal support (Ubuntu14.04 passed the test)

Import reimport loggingimport socketimport jsonimport tracebackimport datetimeimport timetry: from collections import OrderedDictexcept ImportError: percent = ('args', 'asctime', 'created ', 'exc _ info ', 'exc _ text', 'filename', 'funcname', 'levelname', 'levelno', 'lineno', 'module', 'msecs', 'message ', 'msg ', 'name', 'pathname', 'process', 'processname', 'relativecreated', 'stack _ info ',' Thread ', 'threadname') RESERVED_ATTR_HASH = dict (zip (RESERVED_ATTRS, RESERVED_ATTRS) COLORS = {'header':' \ 033 [95m', 'info ': '\ 033 [94m', 'debug':' \ 033 [92m', 'warning': '\ 033 [93m', 'error': '\ 033 [91m ', 'enabled': '\ 033 [0m',} def merge_record_extra (record, target, reserved = RESERVED_ATTR_HASH): for key, value in record. _ dict __. items (): if (key not in reserved and not (hasattr (key, "startswith") D key. startswith ('_'): target [key] = value return targetdef get_host_info (): host_name = ''local_ip = ''try: host_name = socket. gethostname () local_ip = socket. gethostbyname (host_name) Does T Exception, e: pass return host_name, local_ipclass JsonFormatterBase (logging. formatter): def _ init _ (self, * args, ** kwargs): logging. formatter. _ init _ (self, * args, ** kwargs) self. _ required_fields = self. Parse () self. _ skip_fields = dict (zip (self. _ required_fields, self. _ required_fields) self. _ skip_fields.update (RESERVED_ATTR_HASH) def parse (self): standard_formatters = re. compile (R '\((. + ?) \) ', Re. IGNORECASE) return standard_formatters.findall (self. _ fmt) def add_fields (self, record): log_record ={} for field in self. _ required_fields: log_record [field] = record. _ dict __. get (field) host_name, local_ip = get_host_info () log_record [U' @ hostName '] = host_name log_record [U' @ localIp'] = local_ip return log_record # records (record, log_record, reserved = self. _ skip_fields) def process_log_record (self, log_record): "" Override this method to implement custom logic on the possibly ordered dictionary. "try: new_record = OrderedDict () Partition t Exception, e: return log_record key_list = ['asctime', 'levelname', '@ hostname',' @ localip ', 'threadname', 'thread', 'name', 'pathname', 'lineno', 'message',] for k in key_list: new_record [k] = log_record.get (k) new_record.update (log_record) return new_record def jsonify_log_record (self, log_record): "" Returns a json string of the log record. "" return json. dumps (log_record, ensure_ascii = False) def format_col (self, message_str, level_name): "" required color "" return message_str def formatTime (self, record, datefmt = None): ct = self. converter (record. created) if datefmt: s = time. strftime (datefmt, ct) else: t = time. strftime ("% Y-% m-% d % H: % M: % S", ct) s = "% s. % 03d "% (t, record. msecs) return s def format (self, record): if isinstance (record. msg, dict): record. message = record. msg elif isinstance (record. msg, list) or isinstance (record. msg, tuple): record. message = record. msg elif isinstance (record. msg, basestring): record. message = record. getMessage (). split ('\ n') elif isinstance (record. msg, Exception): record. message = traceback. format_exc (record. msg ). split ('\ n') else: record. message = repr (record. msg) if "asctime" in self. _ required_fields: record. asctime = self. formatTime (record, self. datefmt) # if record. exc_info and not message_dict.get ('exc _ info'): # message_dict ['message'] = traceback. format_exception (* record. exc_info) log_record = self. add_fields (record) log_record = self. process_log_record (log_record) message_str = self. jsonify_log_record (log_record) message_str = self. format_col (message_str, level_name = record. levelname) return message_strclass ConsoleFormater (JsonFormatterBase): def _ init _ (self, * args, ** kwargs): JsonFormatterBase. _ init _ (self, * args, ** kwargs) def format_col (self, message_str, level_name): if level_name in COLORS. keys (): message_str = COLORS. get (level_name) + message_str + COLORS. get ('enabled') return message_str def jsonify_log_record (self, log_record): return json. dumps (log_record, ensure_ascii = False, indent = 4) class JsonFileFormater (JsonFormatterBase): def _ init _ (self, * args, ** kewars): JsonFormatterBase. _ init _ (self, * args, ** kewars) def jsonify_log_record (self, log_record): return json. dumps (log_record, ensure_ascii = False)

Configuration

In many cases, we do not implement some Handler, Formater, and other code by ourselves. We can use the config provided by logging to do this. How to Write the config Below is an example to explain,

SC_LOGGING_CONF = {  "version": 1,  "disable_existing_loggers": False,  "formatters": {    "simple": {      "format": "%(asctime)s [%(levelname)s] [%(threadName)s:%(thread)d] [%(name)s:%(lineno)d] - %(message)s"    }  },  "handlers": {    "console": {      "class": "logging.StreamHandler",      "level": "DEBUG",      "formatter": "simple",      "stream": "ext://sys.stdout"    },    "info_file_handler": {      "class": "logging.handlers.RotatingFileHandler",      "level": "INFO",      "formatter": "simple",      "filename": PATH + "info-" + date.today().isoformat() + ".log",      "maxBytes": 10485760,      "backupCount": 20,      "encoding": "utf8"    },    "error_file_handler": {      "class": "logging.handlers.RotatingFileHandler",      "level": "ERROR",      "formatter": "simple",      "filename": PATH + "errors-" + date.today().isoformat() + ".log",      "maxBytes": 10485760,      "backupCount": 20,      "encoding": "utf8"    }  },    "": {      "level": "INFO",      "handlers": ["console", "info_file_handler", "error_file_handler"]    }  }}

First define a formater called simaple, and then define three Handler, which are output to the console, output to the file and info, error.

Logging. config. dictConfig (CONFIG. SC _LOGGING_CONF)
This statement can make these configurations work, which is also done by config. py. You can customize the log without writing a lot of code ..

The above is all the content of this article, hoping to help you learn.

Articles you may be interested in:

• Details about logging in Python
• How to export logs to the console and files simultaneously using Python
• Solve Memory leakage caused by misuse of the logging module in Python
• Introduction to Python logging module logging
• Use the logging module in Python to print log details
• Example of using the python logging class library
• Example of using the logging module in Python
• Use the logging module in Python instead of print (simple logging Guide)
• Python logging module learning notes
• Using the sys template and the logging module in python to obtain the row number and function name
• Example of logging location change in python
• How to use logging in the python standard Log Module
• Python uses the logging and decorator modes to optimize log output.