Tomcat accesslog Log Extension

As a result of the work required, recent research on Tomcat's logs has found that its logs can be broadly divided into two categories, one for running logs, That's what we call the Catalina.out log, which is printed by the tomcat internal code called logger, and the Accesslog access log, which is the information that records external requests for access. With these two types of logs, Tomcat defaults to a different approach,  Run the class log by default is the Java.util.logging framework, the Conf under the logging.properties responsible for configuration management, you can also support switching to LOG4J2 (see my previous post: Upgrade TOMCAT7 The running log framework to LOG4J2 For access logs, Tomcat by default is written directly into the file by date, and is managed by the Server.xml configuration valve.

By default, valve is open and in server.xml we can find the following configuration:

 <  valve  classname  = "Org.apache.catalina.valves.AccessLogValve"   directory  = "Logs"   pattern  = "%h%l%u%t &quot;%r&quot;%s%b"   prefix  = "Localhost_access_log."   suffix  = ". txt"  />  

This configuration generates a Localhost_access_log under logs. Date . txt, which records each external access to some information, the content of the information is configured according to the pattern, the% plus different letters to indicate different information, such as the above default pattern configuration will record "Access IP username time First line request content HTTP status code send byte size , detailed configuration details can refer to Tomcat's Accelog (URL:https://tomcat.apache.org/tomcat-7.0-doc/config/valve.html#Access_ Logging )

By parsing the Accesslogvalve code, its internal Java method operates the file processing code:

@Override Public voidLog (Request request, Response Response,LongTime ) {        if(!getstate (). isavailable () | |!getenabled () | | logelements = =NULL|| Condition! =NULL&&NULL!=request.getrequest (). getattribute (condition)|| Conditionif! =NULL&&NULL==request.getrequest (). getattribute (conditionif)) {            return; }        /*** XXX This was a bit silly, but we want to has start and stop time and * duration consistent.  It would is better to keep start and stop * simply in the request and/or response object and remove time *         (duration) from the interface. */        LongStart =request.getcoyoterequest (). GetStartTime (); Date Date= GetDate (start +Time ); //character BuffersChararraywriter result =Chararraywriters.pop (); if(Result = =NULL) {result=NewChararraywriter (128); }        //different% in pattern means different logelement, where you use result to collect all the additions from the Logelement         for(inti = 0; i < logelements.length; i++{logelements[i].addelement (result, date, request, response, time); }        //writing a file writes result tolog (result); if(Result.size () <=maxlogmessagebuffersize)            {Result.reset ();        Chararraywriters.push (result); }    }

where log (result) is implemented as follows:

@Override Public voidlog (Chararraywriter message) {//check every second to see if you need to switch filesrotate (); //If there is a file, close and reopen a new date file        if(checkexists) {synchronized( This) {                if(Currentlogfile! =NULL&&!currentlogfile.exists ()) {                    Try{Close (false); } Catch(Throwable e) {exceptionutils.handlethrowable (e); Log.info (Sm.getstring ("Accesslogvalve.closefail"), E); }                    /*Make sure date is correct*/Datestamp=Filedateformatter.format (NewDate (System.currenttimemillis ()));                Open (); }            }        }        //log This message synchronously locks to the log file, where buffer is used        Try {            synchronized( This) {                if(Writer! =NULL) {Message.writeto (writer); Writer.println (""); if(!buffered)                    {Writer.flush (); }                }            }        } Catch(IOException IoE) {Log.warn (sm.getstring ("Accesslogvalve.writefail", Message.tostring ()), IOE); }    }

As you can see from the core code above, the default tomcat is to use buffered write files for access logging, if you need to analyze the access log, such as how many IP access in a day, or how many times an IP has been accessed within a minute, The general process is to read the contents of the Accesslog file and analysis, so that on the one hand is unable to meet the purpose of real-time analysis, the more important the amount of data can seriously affect the analysis efficiency, so we need to expand it, for example, we can call the access log into the Kafka or Mango.

The expansion of the previous version of Tomcat 8 was a bit cumbersome compared to the 8 and later versions, because after Tomcat 8, Accesslog was specifically extracting an abstract class that was responsible for assembling the content according to pattern and leaving the log (Chararraywriter message Abstract methods are used to extend, development as long as the extension overrides this method, but the previous version of 8 requires itself to inherit the valvebase and implement the Accesslog interface, rewrite the log (request request, Response Response, long time) method , since the author's company is currently using TOMCAT7 on the line, the following is mainly about how to Accesslog log in Tomcat 7 to expand into Kafka. Steps to Extend:

1. Create lekafkaaccesslogvalve inherit valvebase and implement Accesslog interface:

   @Override Public voidLog (Request request, Response Response,LongTime ) {        if(Producerlist! =NULL&& getenabled () && getState (). IsAvailable () &&NULL!= This. Accesslogelement) {            Try{getnextproducer (). Send (Newproducerrecord<byte[],byte[]> (Topic, This. Accesslogelement.buildlog (Request,response,time, This). GetBytes (Standardcharsets.utf_8)). Get (Timeoutmillis, timeunit.milliseconds); } Catch(Interruptedexception | executionexception |timeoutexception e) {Log.error ("Accesslog in Kafka exception", E); }        }    }

2. Handle the parameters that can be furnished

   PrivateString topic; PrivateString bootstrapservers; //If set to zero then the producer would not wait for any acknowledgment from the server at all.    PrivateString ACKs; PrivateString producersize; PrivateString Properties; Privatelist<producer<byte[],byte[]>>producerlist; PrivateAtomicinteger Producerindex =NewAtomicinteger (0); Private intTimeoutmillis; Private BooleanEnabled =true;//The default configuration asks True, which is to break into Kafka unless there is an exception or an active setting.         PrivateString pattern; Privateaccesslogelement accesslogelement; PrivateString Localename; PrivateLocale locale =Locale.getdefault ();

3. Resolves what needs to be printed according to different pattern configurations (This section tomcat8 has been extracted in abstractaccesslogvalve)

    Public Staticaccesslogelement Parsepattern (String pattern) {Finallist<accesslogelement> list =NewArraylist<>(); BooleanReplace =false; StringBuilder buf=NewStringBuilder ();  for(inti = 0; I < pattern.length (); ++i) {CharCH =Pattern.charat (i); if(replace) {if(' {' = =ch) {StringBuilder name=NewStringBuilder (); intj = i + 1;  for(; (J < Pattern.length ()) && ('} '! = Pattern.charat (j)); ++j) {Name.append (Pattern.charat (j)); }                    if(j + 1 <pattern.length ()) {                        ++J;                        List.add (Createaccesslogelement (name.tostring (), Pattern.charat (j))); I=J; } Else{list.add (createaccesslogelement (ch)); }                } Else{list.add (createaccesslogelement (ch)); } Replace=false; } Else if(ch = = '% ') {Replace=true; List.add (Newstringelement (buf.tostring ())); BUF=NewStringBuilder (); } Else{buf.append (ch); }        }        if(Buf.length () > 0) {List.add (Newstringelement (buf.tostring ())); }        return Newaccesslogelement () {@OverrideprotectedString Buildlog (Request request, Response Response,LongTime , Accesslog Accesslog) {StringBuilder Sbuilder=NewStringBuilder (30);  for(accesslogelement accesslogelement:list) {sbuilder.append (Accesslogelement.buildlog (Request, RE                Sponse, Time, Accesslog)); }                returnsbuilder.tostring ();    }        }; }

4. Adding Configuration in Server.xml

    <  valve  classname  = "Com.letv.shop.lekafkavalve.LeKafkaAccesslogValve"   enabled  = "true"   topic  = "info"   pattern  = "%{yyyy-mm-dd hh:mm:ss}t| | info| | accessvalve| | tomcat| | %a| | %a| | %r| | %s| |%d "  Bootstrapservers  = "Kafka address"   Producersize  = "5"   properties  = "acks=0| | Producer.size=3 " />  

   Tomcat8 and later versions of the extension to be more convenient, directly inherit the abstractaccesslogvalve and rewrite the log method.

Annex is the Kafka implementation of the source download

Tomcat accesslog Log Extension