Elk Series ~nxlog Log collection plus forwarding (resolves a JSON conversion failure caused by LOG4 journal wrapping).

This article will inherit the previous article, mainly through the use of tools to collect and send logs, "Elk series ~nlog.targets.fluentd arrived how to send to Fluentd via TCP"

Nxlog is a log collection tool that locates the system log, or the specified log file, the wildcard character file, and then processes it and finally sends it to the target location. And there are many kinds of target location, such as file system, FLUENTD system, etc., below we introduce a usage scene, is also often involved in the scene.

1. Log4 generation date, date file name, unified suffix, named by log level
2. Nxlog Tools, configure, start, send data
3. FLUENTD configuration, accept data, print data

Log4 generation date, date file name, unified suffix, named by log level

<?xml version="1.0"?><configuration> <configSections>      <section name="log4net"Type="log4net. Config.log4netconfigurationsectionhandler,log4net"/> </configSections> <system.web> <compilation debug="true"targetframework="4.0"/> </system.web> <log4net> <logger name="errorlog"> <level value="ERROR"></level> <appender-ref ref="errorlog"></appender-ref> </logger> <appender name="errorlog"Type="log4net. Appender.rollingfileappender"> <param name="File"Value="logs\\"/> <param name="Appendtofile"Value="true"/> <param name="Staticlogfilename"Value="false"/> <rollingstyle value="Date"/> <datepattern value="yyyymmdd&quot; error.log&quot;"/> <layout type="log4net. Layout.patternlayout"> </layout> </appender> <logger name="InfoLog"> <level value="INFO"></level> <appender-ref ref="InfoLog"></appender-ref> </logger> <appender name="InfoLog"Type="log4net. Appender.rollingfileappender"> <param name="File"Value="logs\\"/> <param name="Appendtofile"Value="true"/> <param name="Staticlogfilename"Value="false"/> <rollingstyle value="Date"/> <datepattern value="yyyymmdd&quot;info.log&quot;"/> <layout type="log4net. Layout.patternlayout"> </layout> </appender> <logger name="Warnlog"> <level value="Warn"></level> <appender-ref ref="Warnlog"></appender-ref> </logger> <appender name="Warnlog"Type="log4net. Appender.rollingfileappender"> <param name="File"Value="logs\\"/> <param name="Appendtofile"Value="true"/> <param name="Staticlogfilename"Value="false"/> <rollingstyle value="Date"/> <datepattern value="yyyymmdd&quot; warn.log&quot;"/> <layout type="log4net. Layout.patternlayout"> </layout> </appender> <logger name="Fatallog"> <level value="Fatal"></level> <appender-ref ref="Fatallog"></appender-ref> </logger> <appender name="Fatallog"Type="log4net. Appender.rollingfileappender"> <param name="File"Value="logs\\"/> <param name="Appendtofile"Value="true"/> <param name="Staticlogfilename"Value="false"/> <rollingstyle value="Date"/> <datepattern value="yyyymmdd&quot; fatal.log&quot;"/> <layout type="log4net. Layout.patternlayout"> </layout> </appender> </log4net></configuration>

View Code

The method of generating the log is as follows

            Log4net. Config.XmlConfigurator.Configure (new FileInfo ("log4.config"));              for (int0; i++)            {                Loghelper.info ("test");            }            Console.WriteLine ("end");

The above code will generate a log file, name the more fixed, method log collection work.

Two Nxlog tools, configure, start, send data

Installation: Https://docs.fluentd.org/v0.12/articles/windows

Configuration: Removed the \ r flag after each line of the LOG4 generated log, if not processed, there will be a problem with JSON conversion

# # this isa sample configuration file. See the Nxlog Reference manual about the## configuration options. It should be installed locally and isalso available## online at http://nxlog.org/docs/# # pleaseSetThe ROOT to the folder your nxlog is installed into,## otherwise it would not start.#defineROOT C:\Program Files\nxlogdefine ROOT C:\Program Files (x86) \nxlogmoduledir%root%\modulescachedir%root%\datapidfile%root%\data\nxlog.pidspooldir%root%\datalogfile%root%\data\nxlog.log<extension json>Module Xm_json</extension><extension _syslog>Module Xm_syslog</extension><inputinch>Module im_file File"C:\dotnet\20*.log"Exec $raw _event= Replace ($raw _event,"\ r \ n"," "); Exec $raw _event= Replace ($raw _event,"\ r"," "); Exec $raw _event= Replace ($raw _event,"\ n"," "); Exec $raw _event= Replace ($raw _event,"0x0A"," "); Exec $raw _event= Replace ($raw _event,"0x0da"," "); Exec $raw _event= Replace ($raw _event,"0x0D"," "); </input><output out>Module om_tcp Host192.168.200.214Port24224Exec $raw _event= $raw _event +"\ n";</output><route1>Pathinch= out</Route>

View Code

Start: Nxlog-f-C conf/nxlog.conf

View: Open the Fluentd side, view its logs, and discover that our logs are here.

This thing, I have been looking for a long time on Google, finally it is Kung fu, let me find the right! It also proves that the original conjecture is correct, that is, filtering the characters in output!

Three FLUENTD configuration, accept data, print data

1 run the script, upgrade the required folders and files

MKDIR/SCRIPTS/FLUENTD-PCD/scripts/Fluentdcat> Dockerfile <<'EOF'From fluent/fluentd:v0. A-onbuildenv TZ=asia/Shanghairun Echo"Http://mirrors.aliyun.com/alpine/v3.5/main">/etc/apk/repositories&& Echo"http://mirrors.aliyun.com/alpine/v3.5/community">>/etc/apk/repositories&& apk Add--Update Tzdata&&apk Add Curl&& ln-snf/usr/share/zoneinfo/$TZ/etc/localtime&& echo $TZ >/etc/timezonerun apk Add--Virtual. build-deps sudo build-Baseruby-Dev&& sudo gem sources--clear- All&& apk del. build-Deps&& RM-RF/var/cache/apk/* \
/home/fluent/.gem/ruby/2.3.0/cache/*.gemeofmkdir plugins-pmkdir-p/srv/volume/fluentd/cat >/scripts/fluentd/ fluent.conf << ' EOF ' <source> @type forward port 24224 bind 0.0.0.0</source><match **> @type s Tdout
</match>eof 

2 Creating a mirror

Docker build--no-cache--pull-t pilipa/tools/fluentd.

2 Docker running directly on

Docker run--privileged=true -v/scripts/fluentd/fluent.conf:/fluentd/etc/fluent.conf pilipa/tools/fluentd

Configuration and log-related output information log collection after startup it's finally over! Thank you for reading! 1024 Happy Holidays!

Elk Series ~nxlog Log collection plus forwarding (resolves a JSON conversion failure caused by LOG4 journal wrapping).