Transform Apache Open source log project to implement distributed log collection system

Overview:

in distributed systems, it is often necessary to collect the logs of each node and then unify the analysis.

This article provides a simple solution, this article uses the Open source log project + the unified database structure way, In each development environment, provide a unified configuration and invocation methods, all logs are recorded in the log server, you can trace the query on any one of the system nodes arbitrary application of the operation of any thread.

Study the current popular Apache open source log Project log4j and its derivative products on other platforms (Log4net log4py, etc.). It outputs logs from the Appender module to different destinations.

For example, using Jdbcappender in log4j can basically implement the function of inserting a database. LOG4J can provide the following data:

Log information,

Log level,

Time

Thread Name,

File path,

Class path,

Line number,

Method name

These data are already well positioned in a single client module to locate the various needs information that the log has occurred. But for our distributed log collection, we still lack the positioning of the machine. As well as application, thread positioning.

For example, running two identical applications on the same machine, recording in the same time log system, we cannot differentiate on the basis of existing lo4j.

It may be said that the use of log4j NDC--NDC is not good, its identification is a string, and the use of NDC words will increase the complexity of the application.

So how do we locate machines, applications, threads? Corresponding to the three we can call the system function to get hostname, ProcessID (process number), ThreadId (thread number)

But in the log4j these three are unable to obtain through the configuration, how solves.

1. Then a very simple idea, we can encapsulate its logger debug, info, warn and other interfaces to achieve. This is a very fucked up place, after you encapsulate, log4j log when the location information (log-initiated time, thread, etc.) becomes our own package position, so it doesn't work.

2. Next we naturally think of rewriting appender, such as the way we rewrite Jdbcappender's write-SQL statements, adding the three variables we need. But this problem will be found: log4j write log is controlled by a task queue thread, it may be correct to get hostname and ProcessID in this queue, but getting the thread ID is definitely wrong. So it won't work.

3. So very helpless, we can only modify the source code of the log4j. We'll find all of the logging methods for the Logger class, which, after passing the license checksum, will call a ForceLog function (which looks like the name), a new Logingevent object, where we modify the constructor, where the PID and thread IDs are passed in, Then inherit jdbcappender, rewrite the function of the spelling SQL statement, and fix it.

Similarly, you can do so in log4net (the architecture is basically similar).

This allows us to obtain our own log jar packages and DLLs, after which we can log the logs according to the original configuration method and the fully compatible log4j and Log4net methods. It also provides our own proprietary Appender, which can be used as a gateway to our distributed log collection system.

Expand the topic:

1. About the size of the log library is too large to retrieve the efficiency is very low how to deal with.

Scenario 1, you can use Lucene to extract information from the log library, which is designed for retrieval.

Scenario 2, periodically clean up and back up the database.

2. Scenarios that apply to this scenario

This scheme is suitable for medium scale (such as 1000 concurrent application nodes), and the node has a stable and fast communication channel (such as LAN), the real-time requirements, record integrity requirements are not very harsh system.

3. Why not use socket-related Appender

After understanding, log4j and its various derivatives actually in the TCP Application layer protocol is not unified, hence not.

4. Copyright License.

Apache license, suitable for business, if you release the code you modified, you need to do the following things:

1. You need to give the user an Apache license when publishing.

2. You need to specify in the original code file where you have modified.

3. The publication requires an agreement, a trademark, a patent statement, and everything in the original project that requires derivative software/class library declarations.

4. The release needs to contain a notice file containing the Apache license and the content of your own claim. (This content cannot conflict with the Apache protocol itself)

Related Code and demo:

LOG4J Configuration

Log4j.rootlogger=debug, DATABASE

Log4j.appender.database=com.netvideo.log.nvjdbcappender

Log4j.appender.database.layout=org.apache.log4j.patternlayout

Log4j.appender.database.url=jdbc:mysql://192.168.25.156:3306/test?useunicode=true&characterencoding=utf-8

Log4j.appender.database.driver=com.mysql.jdbc.driver

Log4j.appender.database.user=xxx

Log4j.appender.database.password=xxxx

Log4j.appender.database.tablename=log

Log4j.appender.database.consoleprint=true

Use method (exactly the same as log4j):

Logger Logger = Logger.getlogger (Test.class);

Logger.info ("info called");

Nvjdbcappender Source

(There's a little episode where Jdbcappender is flawed.) The message cannot be enclosed in single quotes, or it can cause SQL spelling errors, which I have fixed here.

Package com.netvideo.log; Import Org.apache.log4j.jdbc.JDBCAppender; Import java.net.InetAddress; Import java.sql.SQLException; Import Java.text.SimpleDateFormat; Import Java.util.Date; Import Java.util.Iterator; Import Org.apache.log4j.spi.ErrorCode; Import org.apache.log4j.spi.LoggingEvent; /** * Netvideo Common Log module * @author Chenggong * @version 0.1 * * Public class Nvjdbcappender extends the jdbcappender {private stri ng hostName = null; Host name private String tablename; Log table name Private Boolean consoleprint = false;//whether to print public nvjdbcappender () {super () on the console; This.setlocationinfo (true); try {InetAddress ia = inetaddress.getlocalhost (); hostName = Ia.gethostname ();} catch (Exception e) {hostName = "get Exception";}} Protected string Sqlsafe (string s) {return s.replace ("'", ' "');} Protected string GetCurrentTime () {return new string (New SimpleDateFormat ("Yyyy-mm-dd HH:mm:ss"). Format (new Date ()); Protected string getlogstatement (Loggingevent logevent) {String rst = String.Format ("INSERT into%s valuES (null, '%s ', '%s ', '%s ', '%s ',%s, '%s ', '%s ',%d,%d, '%s ', '%s ') ', This.tablename, Logevent.getlevel (), Logevent.getlocationinformation (). GetClassName (), Logevent.getlocationinformation (). GetFileName (), Logevent.getlocationinformation (). Getmethodname (), Logevent.getlocationinformation (). GetLineNumber (), This.sqlsafe (This.hostname), This.sqlsafe (Logevent.getmessage (). toString ()), Logevent.getprocessid (), Logevent.getthreadid (), This.sqlsafe (Logevent.getthreadname ()), this.getcurrenttime ()//pending discussion); return rst; } protected string Getconsoleout (Loggingevent logevent) {String rst = String.Format ("[%s][%s]:%s", This.getcurrenttime ( ), Logevent.getlocationinformation (). Fullinfo, Logevent.getmessage ()); return rst; @SuppressWarnings ("unchecked") public void Flushbuffer () {removes.ensurecapacity (Buffer.size ()); for (Iterator i = buf Fer.iterator (); I.hasnext ();) {try {loggingevent logevent = (loggingevent) i.next (); String sql = getlogstatement (logevent); if (!sql.equals (")) {execute (SQL);} if (cOnsoleprint) {System.out.println (This.getconsoleout (LogEvent));} removes.add (LogEvent); catch (SQLException e) {errorhandler.error ("Failed to Excute SQL", E, errorcode.flush_failure);} Remove from the ' buffer any events ' that were reported buffer.removeall (removes); Clear the buffer of the reported events Removes.clear (); public void Settablename (string tablename) {this.tablename = tablename;} public void Setconsoleprint (string isprint) { Consoleprint = (isprint.tolowercase (). Trim (). Equals ("true")); }} 

Log4net configuration:

<log4net debug= "false" > <root> <level value= "Debug"/> <appender-ref "DB" ref= </root> & Lt;appender name= "DB" type= "Netvideo. Log.netvideoappender "> <buffersize value=" "/> <param name=" ConnectionType "value=" MySql.Data.MySqlClient.MySqlConnection, Mysql.data "/> <param name=" ConnectionString "value=" database=test; Server=192.168.25.156;uid=root;password=123456;old syntax=yes "/> </appender> </log4net>

How to use:

Log4net. ILog logger = log4net. Logmanager.getlogger (System.Reflection.MethodBase.GetCurrentMethod (). DeclaringType);

Logger. Debug ("test");

Netvideoappender Code:

Using System; Using System.Collections; Using System.Data; Using System.IO; Using System.Reflection; Using System.Text; Using Log4net. Appender; Using Log4net. Util; Using Log4net. Layout; Using Log4net. Core; Namespace Netvideo. Log {///<summary>///network video log appender///</summary> public class Netvideoappender:adonetappender {public Netvideoappender () {//console.writeline ("Netvideoappender created");} private string Sqlsafe (string s) {return s.repla CE ("'", "" "). Replace ("//", "////"); } protected void Formatsql (IDbCommand cmd, loggingevent e) {StringBuilder sb = new StringBuilder (); sb. AppendFormat ("INSERT into log VALUES" (null, ' {0} ', ' {1} ', ' {2} ', ' {3} ', {4}, ' {5} ', ' {6} ', {7},{8}, ' {9} ', ' {} '] ', E.level, E.locationinformation.classname, this. Sqlsafe (E.locationinformation.filename), E.locationinformation.methodname, E.locationinformation.linenumber, this . Sqlsafe (E.username), this. Sqlsafe (E.messageobject.tostring ()), E.processid, E.threadid, this. Sqlsafe (e.threadname), e.timestAMP); Cmd.commandtext = sb. ToString (); } override protected void Sendbuffer (IDbTransaction DbTran, loggingevent[] events) {if (M_usepreparedcommand) {//Send B Uffer using the prepared Command object if (M_dbcommand!= null) {if (DbTran!= null) {m_dbcommand.transaction = DbTran; //Run for all events foreach (Loggingevent e in events) {Formatsql (M_dbcommand, E);//Execute the query M_dbcommand. ExecuteNonQuery (); }} else {//Create a new command using (IDbCommand dbcmd = M_dbconnection.createcommand ()) {if (DbTran!= null) {DbC Md. Transaction = DbTran; //Run for all events foreach (Loggingevent e in events) {Formatsql (dbcmd, E); Dbcmd.executenonquery ();}} {}}}}