A solution to memory leaks caused by misuse of logging modules in Python

A solution to memory leaks caused by misuse of logging modules in Python

This article mainly introduces to solve the memory leak caused by the misuse of logging module in Python, because of the problem caused by too much UDP connection, the friend need to refer to the

First introduced how to find it, the online project log is through the logging module to the Syslog, ran a period of time after the discovery of the syslog UDP connection over 8W, yes is 8 W. The main thing is that the logging module is used incorrectly.

We had such a requirement, that is, for each connection log output the current connection information, so each connection creates a log instance, and assigns a Formatter, creates the log instance in order to differentiate the other connections so I'm simply rude. Use the ID of the current object as the journal name:

?

1 2 3 4 5 6 7

Import Logging Class Connection (object): Def __init__ (self): Self._logger_name = "Connection. {} '. Format (ID (self)) Self.logger = Logging.getlogger (self._logger_name)

Of course, the test environment is open debug, debug will not go to the syslog, so there will not be too many UDP connections, you will not know that there are memory leaks, let's see why this leads to memory leaks, first look at the GetLogger code:

?

1 2 3 4 5 6 7 8 9 10

def getlogger (Name=none): "" "return a logger with the specified name, creating it if necessary. If No name is specified, return the root logger. ' "' If Name:return Logger.manager.getLogger (name) else:return root

The main call to Logger.manager.getLogger, this function has the following section of code fragment

?

1 2 3 4 5 6 7 8 9 10 11 12 13-14

If name in SELF.LOGGERDICT:RV = Self.loggerdict[name] If isinstance (rv, placeholder): ph = RV RV = (Self.loggerclass or _ Loggerclass) (name) Rv.manager = self Self.loggerdict[name] = RV Self._fixupchildren (ph, RV) self._fixupparents (RV) Else: RV = (Self.loggerclass or _loggerclass) (name) Rv.manager = self Self.loggerdict[name] = RV self._fixupparents (RV)

Logging module to ensure that the same name refers to the same log instance, so that all the log instances exist in a loggerdict dictionary, so unless the program exits, the creation of the log instance reference is not released, so the log instance of the handlers will not be released. The ID of the object I used before was part of the log name, so the UDP connection created by Sysloghandler has been occupied and caused too many UDP connections.

To solve this problem I added the following code when the connection was closed:

?

1 2 3

Logging. Logger.manager.loggerDict.pop (self._logger_name) Self.logger.manager = None Self.logger.handlers = []

Supposedly only add the first line of code should be released, but no, so there is a third line of code, Sysloghandler finally released, the issue is not yet known why, but also need to check.

2015-03-30 Update if the journal name is with. Delimited, the logging module takes the last part as the log name and goes up to look for the parent Logger, and if not found, creates the placeholder object as the parent and references Logger.

For example, the Logger name created is called A.B.C, then the actual name is C, and B is the parent of C, and A is the parent of B, and if a Logger without that name creates a placeholder object instead, placeholder creates the current A reference to the Logger. So the name of the log object that needs to be reclaimed should not be included.