High throughput of a log function class _ for IOCP (Delphi)

In the development of server-side programs, the log is a necessary function. Since the server side of the data to be frequently written to the log, the beginning of the time with a very simple log function

is to write the log characters directly into the file. Then close the connection. has been well applied. But when doing stress testing, because to each connected data to write to the log, found that after a period of time, frequent drop, CPU occupancy, high, optimization can be thought of a lot of places, there is a certain effect, careful observation found that hard flash drive lights not only, the hard drive i/0 operation too tense. But when the test, the basic is not read and write hard disk, suddenly found that the log function affects the performance of the entire system. For each log data, open the file, write the file, and close the file. Ha, these are relatively expensive i/0 operations. The optimization method is simple, caching data, and periodically writing to disk in bulk. Based on this design idea, a new log class is developed. Space in exchange for time.

The internal use of double buffering algorithm, write information, is directly written to memory, and then the thread according to a certain time interval, the memory of the data written to the disk file, which opened two buffer memory queue, using the producer = = "Consumer mode, Writelog is written to the log data, The data producer, TFileStream object, writes in-memory data to disk is the consumer role, due to the use of double buffering method, reducing the interference between production and consumption. Improves performance and reduces log write times. is also barely a practical application of a double-buffered queue.

This log class is based on thread implementation. For ease of use. Locking mechanisms are used internally. is a thread-safe class. When the amount of data is large or to facilitate the management of log files

We will generate different log file names according to certain rules, the most common is the name of the log files by date. such as 20110702.log, 20110701.log, etc.

In this log class, you can change the log file name at any time, given this condition.

Property filename:string read Getlogfilename write setlogfilename; To modify the log file name, directly assign a value. is also thread-safe.

Finally, the design of this log class can also be used for other aspects. Buffering, space-changing time is a common method in software design.

This log class was applied to the server program (ECHO test 12,000 connection OK) that was developed in later IOCP mode. Good results

Code of Implementation

Unit usflog;interfaceuses Windows, Messages, sysutils, variants, Classes;type Tsflog=class (tthread) Private flf:str     ing;//#13 # #;     Fs:tfilestream;     fcurfilename:string;     ffilename:string;     Fbegcount:dword;     Fbuffa,fbuffb:tmemorystream;     Fcs:trtlcriticalsection;     Fcs_filename:trtlcriticalsection;     Flogbuff:tmemorystream;     Procedure WriteToFile ();     function getlogfilename:string;   Procedure Setlogfilename (const value:string);   Protected procedure Execute (); override;      Public constructor Create (logfilename:string);      destructor Destroy (); override;      Procedure Writelog (const inbuff:pointer;insize:integer); overload;  Procedure Writelog (const msg:string); overload;  Public Property filename:string Read Getlogfilename write setlogfilename; end;implementation{tsflog}constructor tsflog.create (logfilename:string); begin if Trim (logfilename) = "Then Rai" Se exception.   Create (' Log FileName not ' '); Inherited Create (TRUE);  \ initializecriticalsection (FCS);   Initialize InitializeCriticalSection (fcs_filename);//log file name//queue buffer B run, alternately use Self.fbuffa: = Tmemorystream.create (); Self.FBuffA.Size: = 1024 * 1024;   The initial value can be adjusted SELF.FBUFFB as needed: = Tmemorystream.create (); Self.FBuffB.Size: = 1024 * 1024;     The initial value can be adjusted Self.flogbuff as needed: = Self.fbuffa;      If FileExists (logfilename) THEN begin FS: = Tfilestream.create (Logfilename,fmopenwrite or fmsharedenywrite); Fs. Position: = FS. Size;   If the file already exists, append the data to the end else FS: = Tfilestream.create (logfilename,fmcreate or fmsharedenywrite);   Fcurfilename: = LogFileName;   Ffilename: = LogFileName;   FLF: = #13 # #;  Start execution Self.resume ();   \end;destructor tsflog.destroy;begin Fbuffa.free ();   Fbuffb.free (); Fs.   Free ();   Inherited;end;procedure Tsflog.execute (); begin Fbegcount: = GetTickCount ();  while (not self.terminated) does begin//2000ms can be adjusted to suit your needs, the data is written to disk at the interval if (GetTickCount ()-Fbegcount) >= 2000 Then BEGIN       WriteToFile ();    Fbegcount: = GetTickCount ();   End Else Sleep (200);    End   WriteToFile (); end;function tsflog.getlogfilename:string;begin entercriticalsection (FCS_FileName);    Try Result: = fcurfilename;    Finally LeaveCriticalSection (fcs_filename);  End;end;procedure tsflog.setlogfilename (const value:string); begin EnterCriticalSection (Fcs_filename);  Try ffilename: = Value;  Finally LeaveCriticalSection (fcs_filename); End;end;procedure tsflog.writelog (const msg:string); Begin Writelog (Pointer (msg), Length (msg)); end;procedure Tsflog.writelog (const inbuff:pointer;    Insize:integer) var tmpstr:string;begin tmpstr: = FormatDateTime (' yyyy-mm-dd hh:mm:ss zzz ', now ());    EnterCriticalSection (FCS);      Try Flogbuff.write (Tmpstr[1],length (TMPSTR));      Flogbuff.write (inbuff^,insize);    Flogbuff.write (flf[1],2);  Finally LeaveCriticalSection (FCS);    End;end;procedure Tsflog.writetofile;var Ms:tmemorystream; IslogfilenaMechanged:boolean;begin entercriticalsection (FCS);      Swap buffer try MS: = nil;          If flogbuff.position > 0 THEN BEGIN MS: = Flogbuff;         If Flogbuff = Fbuffa then flogbuff: = FBUFFB else Flogbuff: = Fbuffa;     Flogbuff.position: = 0;  End  Finally LeaveCriticalSection (FCS);  End   \ if MS = nil then Exit; Write the file to a try FS. Write (MS. Memory^,ms.    Position); Finally MS.    Position: = 0;     End    Detects if the file name changes entercriticalsection (fcs_filename);    Try islogfilenamechanged: = (fcurfilename <> ffilename);    Finally LeaveCriticalSection (fcs_filename);     End        The log file name modifies if islogfilenamechanged then begin fcurfilename: = Ffilename; Fs.       Free ();         If FileExists (ffilename) THEN begin FS: = Tfilestream.create (Ffilename,fmopenwrite or fmsharedenywrite); Fs. Position: = FS.     Size; End else FS: = Tfilestream.create (ffilename,fmcreate or fmsharedenywrite);    End;end;end. Test code for Log class function//main test three functions 1) is the log writing speed fast enough 2) the log class can run stably in multi-threaded situations? 3) in operation, replace the file name of the output log in order to generate a large number of data and stability under multi-threading, the test generated 120 threads, while writing log functions. At the same time with a timer, periodically modify the output log file name.  The information written to the log is a randomly generated GUID string. Unit umain;interfaceuses Windows, Messages, sysutils, variants, Classes, Graphics, Controls, Forms, Dialogs,usflog, StdC    TRLs, Extctrls,activex;type Tfrmmain = Class (Tform) Button1:tbutton;    Edit1:tedit;    Timer1:ttimer;    Procedure Formcreate (Sender:tobject);    Procedure Button1Click (Sender:tobject);    Procedure Formclose (Sender:tobject; var action:tcloseaction);  Procedure Timer1timer (Sender:tobject);    Private {private declarations} flist:tlist;  Logobj:tsflog;  Public {public declarations} end;  Tsflogtest=class (TThread) protected procedure Execute (); override;  End;var frmmain:tfrmmain;implementation{$R *.dfm}function getguid (): String;var id:tguid;begin CoCreateGuid (ID); Result: = guidtostring (ID); End;procedure tfrmmain.formcreate (Sender: TObject); begin logobj: = Tsflog.create (' C:\temp\0001.TXT '); Flist: = Tlist.create (); end;  Start test procedure Tfrmmain.button1click (sender:tobject); var obj:tsflogtest;    Index:integer;begin for Index: = 1 to + do begin OBJ: = Tsflogtest.create (FALSE);  Flist.add (OBJ);  End Self.Timer1.Enabled: = True;end; {tsflogtest}procedure Tsflogtest.execute;var msg:string;begin.  Terminated) DO begin MSG: = IntToStr (self.threadid) + #09 + getguid () + getguid () + getguid () + GetGuid ()    + getguid () + getguid () + getguid () + GetGuid ();    FrmMain.LogObj.WriteLog (MSG);  Sleep (10);  End;end;procedure tfrmmain.formclose (Sender:tobject; var action:tcloseaction); var Index:integer;    Obj:tsflogtest;begin for Index: = 0 to Flist.count-1 do begin obj:= Tsflogtest (Flist.items[index]);  Obj.terminate ();  End Sleep (+); end;procedure Tfrmmain.timer1timer (Sender:tobject); var afilename:string;begin afilename: = ' C:\Temp\ ' + Fo Rmatdatetime (' Yyyymmdd_hhmmss_zzz ', now ()) + '.  TXT '; Logobj.filename: = Afilename;end;end.

A log function class with high throughput for IOCP (Delphi)