Use the mina2 framework to write distributed log servers

Log Server, the game server is an instance, an important database server, game server, and gateway server... generally, logs are processed by log4j, log4c in C, and log4cpp in CPP. Of course, these log tools are the mainstream open-source frameworks, performance and functionality are good. However, if we need to record almost every player, and there is no operation, such log records added to the application will consume a large amount of system resources, this slows down the server. gu, distributed log server is very meaningful in some project architectures.

My distributed server is divided into three versions: C language, native Java NiO, and dependent on the implementation of the mina2 framework. Due to the rapid development of the mina2 framework, I believe that, this article is helpful to Java programmers.

Please refer to the source code:

Write iohandler. log4j records errors and other logs, while logwriter is a real high-frequency, big data logging tool.

Package CN. vicky. mina. logserver; import Java. util. collections; import Java. util. hashset; import Java. util. set; import Org. apache. mina. core. buffer. iobuffer; import Org. apache. mina. core. service. iohandler; import Org. apache. mina. core. session. idlestatus; import Org. apache. mina. core. session. iosession; import Org. slf4j. logger; import Org. slf4j. loggerfactory;/***** @ author Vicky. H */public class logiohandler implements iohandler {Private Static final logger log = loggerfactory. getlogger (logiohandler. class); private set <iosession> sessions = collections. synchronizedset (New hashset <iosession> (10); // log writer private logwriter; Public logiohandler (logwriter) {This. logwriter = logwriter;} public void sessioncreated (iosession session) throws exception {// do nothing ...} public void sessionopened (iosession session) throws exception {sessions. add (session);} public void sessionclosed (iosession session) throws exception {sessions. remove (session);} public void sessionidle (iosession session, idlestatus status) throws exception {// do nothing ...} public void exceptioncaught (iosession session, throwable cause) throws exception {log. error ("exceptioncaught:", cause);} public void messagereceived (iosession session, object message) throws exception {If (Message instanceof iobuffer) {// confirm the Data Type logwriter. addlog (iobuffer) Message) ;}} public void messagesent (iosession session, object message) throws exception {// do nothing ...}}

 

The core idea of logwriter is to achieve high performance and high concurrency of data storage and writing to files through direct cache and cache swap, and write files in about 5 minutes, when the log time requirement is not very accurate, you can obtain the log range by using the log file name method, which can streamline the log file size and reduce performance consumption. hereSmall files are used to record logs. Another reason is to facilitate data statistics. in terms of log processing, I did not write additional programs, but used regular expressions to disassemble the files. Instead, I used Linux shell commands to disassemble the files.

Package CN. vicky. mina. logserver; import Java. io. file; import Java. io. fileoutputstream; import Java. io. ioexception; import Java. NIO. bytebuffer; import Java. NIO. channels. filechannel; import Java. text. dateformat; import Java. text. simpledateformat; import Java. util. date; import Org. apache. mina. core. buffer. iobuffer; import Org. slf4j. logger; import Org. slf4j. loggerfactory;/***** @ author Vicky. H * log write timer */Public Class logwriter extends thread {Private Static final logger log = loggerfactory. getlogger (logwriter. class); Private Static final dateformat df = new simpledateformat ("yymmddhhmmss"); // format the log time private final object syncobj = new object (); // buffer synchronization lock // direct buffer, which can be better coupled with the current operating system, thus further improving the I/O operation speed. // However, the system overhead of allocating a direct buffer is very large. Therefore, this buffer is used only when the buffer is large and long-lived or requires frequent reuse. // Therefore, using direct buffer is very suitable for private bytebuffer logcontent; private bytebuffer logcontentswap; private Boolean isrunning = true; // The running status of the log thread is private long interval = 180000; // 180 seconds interval private int capacity = 1024*1024*18; // buffer size, reserved enough, about 60 seconds 6 m public logwriter () {This. logcontent = bytebuffer. allocatedirect (capacity); // 10m buffer this. logcontentswap = bytebuffer. allocatedirect (capacity); // 10 m buffer swap zone }/** * Set the log Writing Tool * @ Param capacity buffer size * @ Param interval write interval (preferably less than 5 minutes) * The buffer size must be reserved enough for about 100 seconds. 1 m capacity = interval/10000 */Public logwriter (INT capacity, long interval) {This. logcontent = bytebuffer. allocatedirect (capacity); this. logcontentswap = bytebuffer. allocatedirect (capacity); this. interval = interval;} public long getinterval () {return interval;}/*** the default value is 3 minutes, when not precise * Based on the log time, you can obtain the record time of each log by using the log file name. * @ Param interval */Public void setinterval (long interval) {This. interval = interval;} public void addlog (iobuffer log) {synchronized (syncobj) {// buffer is a non-thread-safe logcontent. put (log. buf (); // write log Content} public void Shutdown () {log.info ("Disable logger"); this. isrunning = false ;}@ override public void run () {log.info ("enable logger"); While (isrunning) {log.info (New Date () + "Write log"); If (logcontent. position () = 0) {log.info (new date () + "no log Content, skip writing to file"); Continue ;}try {thread. sleep (interval);} catch (interruptedexception e) {log. error ("thread sleep exception! ", E);} Try // process tempcontent {file = new file (" C:/log _ "+ DF. format (new date () + ". log "); file. createnewfile (); fileoutputstream Fos = new fileoutputstream (File); filechannel = FOS. getchannel (); synchronized (syncobj) {// set the temporary buffer to point to the currently accepted data buffer bytebuffer tempcontent = logcontent; // The Current Buffer points to the buffer swap zone logcontent = logcontentswap; // The buffer swap area points to the original buffer address logcontentswap = tempcontent;} log Contentswap. flip (); filechannel. write (logcontentswap, logcontentswap. position (); filechannel. close (); FOS. close (); // after processing, reset the buffer logcontentswap. clear ();} catch (ioexception e) {log. error ("An error occurred while writing logs to a file! ", E );}}}}

 

Simple test:

Package CN. vicky. mina. logserver; import Java. util. date; import Java. util. logging; import Org. apache. mina. core. service. ioacceptor; import Org. springframework. context. configurableapplicationcontext; import Org. springframework. context. support. classpathxmlapplicationcontext;/***** @ author Vicky. H */public final class mainapp {public static void main (string ARGs []) {If (system. getproperty ("com. sun. managem Ent. jmxremote ")! = NULL) {system. out. println ("JMX enabled. ");} else {system. out. println ("JMX disabled. please set the "+" 'com. sun. management. jmxremote 'System property to enable JMX. ");} configurableapplicationcontext applicationcontext = new classpathxmlapplicationcontext (" cN/Vicky/Mina/logserver/logservercontext. XML "); system. out. println ("listening... "); ioacceptor = (ioacceptor) applicationcont Ext. getbean ("ioacceptor"); ioacceptor. setcloseondeactivation (false); // set the unbind () method // todo to enable the test class testthread extends thread {@ override public void run () {New testclient (). test () ;}} testthread thread1 = new testthread (); testthread thread2 = new testthread (); testthread thread3 = new testthread (); testthread thread4 = new testthread (); testthread thread5 = new testthread (); thread1.star T (); thread2.start (); thread3.start (); thread4.start (); thread5.start (); system. Out. println ("Enter exit to close the server! "); Response response = new response (system. In); string command = response. nextline (); While (! Command. equals ("exit") {command = response. nextline ();} If (ioacceptor. iscloseondeactivation () {applicationcontext. close (); // close the container system. out. println ("all connections will be closed once unbind ()");} else {// two steps are required to disable the server applicationcontext. close (); system. out. println ("all connections after unbind () is blocked"); ioacceptor. dispose (true); system. out. println ("Disable server");} system. out. println ("the server will be completely shut down in 1 minute! "+ New date ());}}

Test Client

Package CN. vicky. mina. logserver; import java.net. inetsocketaddress; import Java. NIO. charset. charset; import Org. apache. mina. core. filterchain. defaultiofilterchainbuilder; import Org. apache. mina. core. filterchain. iofilter; import Org. apache. mina. core. future. connectfuture; import Org. apache. mina. core. service. ioconnector; import Org. apache. mina. core. service. iohandler; import Org. apache. mina. core. session. idlesta Tus; import Org. apache. mina. core. session. iosession; import Org. apache. mina. core. session. iosessionconfig; import Org. apache. mina. filter. codec. protocolcodecfactory; import Org. apache. mina. filter. codec. protocolcodecfilter; import Org. apache. mina. filter. codec. textline. linedelimiter; import Org. apache. mina. filter. codec. textline. textlinecodecfactory; import Org. apache. mina. transport. socket. NIO. niosocketconnect Or; import Org. slf4j. logger; import Org. slf4j. loggerfactory;/***** @ author Vicky. H */public class testclient {public void test () {final ioconnector conne= new niosocketconnector (); // 2. iosessionconfig sessionconfig = connector. getsessionconfig (); sessionconfig. setreaderidletime (60000); sessionconfig. setreadbuffersize (1024*2); // 3 defaultiofilterchainbuilder filterchain = connector. getfilt Erchain (); protocolcodecfactory codecfactory = new textlinecodecfactory (charset. forname ("UTF-8"), linedelimiter. windows. getvalue (), linedelimiter. windows. getvalue (); iofilter filter = new protocolcodecfilter (codecfactory); filterchain. addlast ("codec", filter); // 5 connector. sethandler (New iohandler () {logger = loggerfactory. getlogger (iohandler. class); Public void sessioncreated (iosession Session) throws exception {logger. debug ("Create a session link (not connected)");} public void sessionopened (iosession session) throws exception {logger. debug ("connections will be established between the server and the client");} public void sessionclosed (iosession session) throws exception {logger. debug ("the server-client session link has been closed");} public void sessionidle (iosession session, idlestatus status) throws exception {logger. debug ("the client enters idle state"); Session. close (false);} public void Exceptioncaught (iosession session, throwable cause) throws exception {logger. error ("error: Exceptions caught by the client", cause); Session. close (true);} public void messagereceived (iosession session, object message) throws exception {logger. debug ("received server message:" + message);}/*** note that this is the callback after the message is sent, not the message to be sent !!! */Public void messagesent (iosession session, object message) throws exception {// disconnect the link from the client after sending the returned message, this is similar to the short link mode like HTTP requests // session. close (true)}); // 6 connectfuture = connector ctor. connect (New inetsocketaddress ("192.168.1.77", 1234); // 7 connectfuture. awaituninterruptibly (); iosession session = connectfutfuture. getsession (); system. out. println ("connection successful! "); For (INT I = 0; I <999; I ++) {session. Write (" test log server performance !!! "+ I); try {thread. sleep (30);} catch (interruptedexception e) {e. printstacktrace () ;}} public static void main (string ARGs []) {New testclient (). test ();}}

 

 

 

About shell disassembly log files:

PID: 72356 item: 11217 num: 1 cast: 2
PID: 72529 item: 11217 num: 1 cast: 2
PID: 52567 item: 262 num: 1 cast: 20
PID: 52567 item: 262 num: 1 cast: 20
PID: 72539 item: 11217 num: 1 cast: 2
PID: 72151 item: 11217 num: 1 cast: 2
PID: 70569 item: 11318 num: 3 cast: 48

As shown in the preceding log file, shell script:

#! /Bin/bash
# Calculate the number of purchases of item: 11218 and the number of purchases according to the log file

For item in 11218 11219 11217 230

Do
Awk '$4 ~ /Item: '$ item'/{count + = 1; sub ("Num:", "", $5 ); sum + = $5} end {print '$ item', Count = ""? "0": "" count "", sum = ""? 0: Sum} 'log.2012-0 * (date range)> log.txt

Done