Flume-ng spooldir distributed Log Collection directory monitoring

OK, come straight to the dry

In the use of Flume-ng, stepped a lot of pits, now for a moment, I hope you bypass the pit to reach the purpose of skilled use of flume

The first pit: can not correctly decode the file, causing the file can not be correctly renamed, after throwing a bug, all files can not be collected by Flume, is a more serious mistake, caused by Flume using NiO method to read the file, will read the file to UTF-8 encoding read, In the Linux state, the default is to store files according to GBK encoding, so the reading will encounter the character length is not enough, workaround: In the Flume configuration file set the monitoring directory to read the encoding of the file.

Second pit: CP large file throws exception, this is because the copy to Spooldir file can not be modified, workaround, use SCP or CP backup file, and then the file MV into Spooldir. You can also use the suffix regular mate to copy the. tmp file, and wait for the copy to complete, rename is not in the first name.

Second pit to modify the source code:

Can modify the source code to avoid large file upload problem, is to modify the source information:

Here are the source code methods that need to be modified:

/** * * @Title: Checkfilecpisover * @Description: Used to check if file copy completed * @param @param currentfile settings file * @ return void returns type * @throws */private void Checkfilecpisover (file file) {Logger.info ("Checkfilecpiso        Ver start ..... "); Long modified = file.lastmodified ();//Current file modification time long length = File.length ();//Current file size try {Threa        D.sleep (1000);//wait 1 seconds} catch (Interruptedexception e) {e.printstacktrace ();        } File Currentfile = new file (File.getabsolutepath ()); int count = 0;//records the number of cycles, more than 20 times, which is 10 seconds after the exception is thrown while (currentfile.lastmodified ()! = Modified | | currentfile.length ()! = Le Ngth) {if (Count >) {String message = "File Copy time too long.                        Please check copy whether exception! "+" \ n "+" File at: "+ file.getabsolutepath () +" \ n "                + "File Current length is:" + currentfile.lastmodified (); New IllegalStateException (message);            } count++;            Modified = Currentfile.lastmodified ();            Length = Currentfile.length (); try {thread.sleep (500);//wait 500 milliseconds} catch (Interruptedexception e) {//TODO Auto            -generated Catch block E.printstacktrace ();        } currentfile = new File (File.getabsolutepath ()); }//Until the file transfer is complete you can exit}

After modification, the jar package is uploaded to Flume to replace the original jar.

Flume-ng spooldir distributed Log Collection directory monitoring