Python log collection server and python log collection

Python log collection server and python log collection
Note: The log collection service of python is thread-safe (locks are used for writing the same file), but it cannot be processed in the case of multiple processes. The Recommended Practice in the official python documentation is to use the tcp server for log collection to ensure secure writing of files. The log collection server is implemented based on twisted for reference. The program is tested on centos and can be used in the production environment.

When the client and server are on one machine, 6000 logs can be processed per second.

#! /Usr/bin/env python #-*-coding: UTF-8-*-################################### logserver # receiving and storing log information through socket # Note: the test program is test_logserver.py # modification time: 2013-11-22 #################################### standard libimport OS, sysimport timeimport twistedimport loggingimport tracebackimport structimport cPickle # use cPickle for speedfrom logging. handlers import TimedRotatingFileHandlerfrom twisted. internet. protocol Import Protocol, Factory # our own codeimport settingsfrom log_record import initlog # generate a def gen_log_name (name): name = name Based on the logger name. replace ('. ',' _ ') return name + '. log 'class ReceiveProtocol (Protocol): HEADER_LENGTH = 4 def _ init _ (self): self. buf = ''self. state = 'init 'self. require_length = ReceiveProtocol. HEADER_LENGTH # record the logger name self. logger_dict ={} def connectionMade (self): Logger = logging. getLogger ('Log _ Server') logger.info ('[makeConnection]) establishes a socket connection with logger_client. ') Def dataReceived (self, data): logger = logging. getLogger ('Log _ Server') try: if len (self. buf + data)> = self. require_length: self. buf = self. buf + data # data ready for corresponding action data = self. buf [0: self. require_length] # Remove Data from the buffer zone self. buf = self. buf [self. require_length:] self. solve (data) # multiple log records may be read at a time if self. buf: self. dataReceived ('') else: self. buf + = data processing t BaseException, e: logger. error ('failed to process SMS record' + Str (e) + '\ n' + traceback. format_exc () def solve (self, data): logger = logging. getLogger ('Log _ Server') statehandler = None try: pto = 'proto _ '+ self. state statehandler = getattr (self, pto) handle T AttributeError: logger. error ('callback', self. state, 'not found ') self. transport. loseConnection () statehandler (data) if self. state = 'done': self. transport. loseConnection () def connectionLost (self, reas On): logger = logging. getLogger ('Log _ Server') logger.info ('[connectionLost] And logger_client socket Connection closed. ') # Record log def proto_record (self, data): logRecord = logging. makeLogRecord (cPickle. loads (data) if logRecord. name not in self. logger_dict: logger = initlog (logRecord. name, settings. PATH, gen_log_name (logRecord. name), logLevel = logging. DEBUG) self. logger_dict [logRecord. name] = logger self. logger_dict [logRecord. name]. handle (logRecord) # modify the next action and the required length self. state = 'init 'self. require_length = ReceivePro Tocol. HEADER_LENGTH # process the header information def proto_init (self, data): length = struct. unpack ('! I ', data [0: ReceiveProtocol. HEADER_LENGTH]) [0] # modify the next action and the required length self. state = 'record 'self. require_length = length if len (self. buf)> = self. require_length: data = self. buf [0: self. require_length] # Remove Data from the buffer zone self. buf = self. buf [self. require_length:] self. solve (data) class ReceiveFactory (Factory): def buildProtocol (self, addr): return ReceiveProtocol () def main (): print 'logserver has started. 'logger = logging. getLogger ('Log _ Server') logger.info ('logserver has started. ') from twisted. internet import epollreactor. install () reactor = twisted. internet. reactor. listenTCP (settings. PORT, ReceiveFactory () reactor. run () if _ name _ = '_ main _': main ()

Completion program:

Https://github.com/vearne/log_server/tree/master

References:

Https://docs.python.org/2/howto/logging-cookbook.html#logging-cookbook

Server logs monitored by scripts in my python language. When errors occur in logs, a voice alarm is triggered Based on the keyword. However, this problem has recently occurred.

Keyword, wav = line. strip (). split.
You can use the following two methods to modify it:
1. only care about the first two items separated by. And change them to the following:
L = list. strip (). split ('.)
Keyword = l [0]
Wav = l [1]
That is, no matter how many items there are, only the first two items are taken
2. focus only on the two items separated by the first.
L = s. find (".")
Keyword = s [0: l]
Wav = s [l + 1:]
You need to determine which of the two Processing Methods meets your program's intention. You can print the line value before 64 rows before modification to determine the print line.

The error log of the python apache django server is always like this. What is the problem?

How can I get a PHP warning?