Solve the memory leakage caused by misuse of the logging module in Python, pythonlogging

Solve the memory leakage caused by misuse of the logging module in Python, pythonlogging

First, let's introduce how to find it. The online project log is sent to syslog through the logging module. After a while, we found that the syslog UDP connection exceeded 8 W, which is 8 W. this is mainly because the logging module is not used properly.

We previously had such a requirement that the current connection information is output for each connection log. Therefore, a log instance is created for each connection and a Formatter is assigned, to distinguish other connections, I simply and roughly used the id of the current object as the log Name:

import loggingclass Connection(object):  def __init__(self):    self._logger_name = "Connection.{}".format(id(self))    self.logger = logging.getLogger(self._logger_name)

Of course, in the test environment, DEBUG is enabled, so DEBUG won't be used in syslog, so there won't be too many UDP connections, and there will be no memory leakage, let's take a look at the cause of Memory leakage. First, let's look at the getLogger code:

def getLogger(name=None):  """  Return a logger with the specified name, creating it if necessary.  If no name is specified, return the root logger.  """  if name:    return Logger.manager.getLogger(name)  else:    return root

Logger. manager. getLogger is called. This function contains the following code snippet:

      if name in self.loggerDict:        rv = self.loggerDict[name]        if isinstance(rv, PlaceHolder):          ph = rv          rv = (self.loggerClass or _loggerClass)(name)          rv.manager = self          self.loggerDict[name] = rv          self._fixupChildren(ph, rv)          self._fixupParents(rv)      else:        rv = (self.loggerClass or _loggerClass)(name)        rv.manager = self        self.loggerDict[name] = rv        self._fixupParents(rv)

To ensure that the same name references the same log instance, the logging module stores all log instances in a loggerDict dictionary, the created log instance reference will not be released, so the handlers in the log instance will not be released. previously, I used the Object id as a part of the log name, so the UDP connection created by SyslogHandler has been occupied, resulting in too many UDP connections.

To solve this problem, I added the following code when the connection is closed:

logging.Logger.manager.loggerDict.pop(self._logger_name)self.logger.manager = Noneself.logger.handlers = []

It is said that only the first line of code above should be released, but no, so with the third line of code, SyslogHandler will be released. This problem is unknown for the time being, check again.

Update if the log name is. the logging module uses the last part as the log name and looks up for the parent Logger. If the parent Logger cannot be found, the PlaceHolder object is created as the parent and the Logger is referenced.

For example, the created Logger is named. b. c, then the actual name is c, and B is the parent of c, a is the parent of B, if there is no Logger with this name, The PlaceHolder object is created as a replacement, placeHolder creates a reference to the current Logger. therefore, the name of the log object to be recycled should not contain.