Logstash 6.x collecting syslog logs

1, Logstash end

Close the rsyslog of the Logstash machine and release the 514 port number

[Root@node1 config]# systemctl stop Rsyslog
[root@node1 config]# systemctl status Rsyslog
Rsyslog.service-sys TEM Logging Service
   loaded:loaded (/usr/lib/systemd/system/rsyslog.service; enabled; vendor preset:enabled)
   Active:inactive (dead) since Thu 2018-04-26 14:32:34 CST; 1min 58s ago
  process:3915 execstart=/usr/sbin/rsyslogd-n $SYSLOGD _options (code=exited, status=0/success)
 Main pid:3915 (code=exited, status=0/success)

Apr 14:25:16 Node1 systemd[1]: Starting System Logging Service...
  APR 14:25:16 node1 systemd[1]: Started System Logging Service.
APR 14:32:34 Node1 systemd[1]: stopping System Logging Service
... APR 14:32:34 node1 systemd[1]: Stopped System Logging Service.
[Root@node1 config]#

Writing the Logstash configuration file

[Root@node1 logstash-6.2.3]# VI config/local_syslog.conf
[root@node1 logstash-6.2.3]# cat Config/local_ syslog.conf
Input {
 syslog {
    type = "Rsyslog"
    port = "514"
  }
}
output{
  stdout{
    codec = Rubydebug
  }
}

Start Logstash

[Root@node1 logstash-6.2.3]# bin/logstash-f config/local_syslog.conf sending Logstash ' s logs to/var/log/logstash which is now configured via log4j2.properties [2018-04-26t14:39:57,627][info][logstash.modules.scaffold] Initializing Module {:module_name=> "NetFlow",:d irectory=> "/opt/logstash-6.2.3/modules/netflow/configuration"} [ 2018-04-26t14:39:57,650][info][logstash.modules.scaffold] Initializing module {:module_name=> "Fb_apache",: Directory=> "/opt/logstash-6.2.3/modules/fb_apache/configuration"} [2018-04-26t14:39:58,301][warn] [ 
Logstash.config.source.multilocal] Ignoring the ' pipelines.yml ' file because modules or command line options is specified [2018-04-26t14:39:59,346] [INFO] [Logstash.runner] Starting Logstash {"Logstash.version" = "6.2.3"} [2018-04-26t14:40:00,022][info][logstash.a Gent] Successfully started Logstash API endpoint {:p ort=>9600} [2018-04-26t14:40:04,438][info][logstash.pip Eline] Starting pipeline {:p IPELINE_ID=&Gt; " Main "," Pipeline.workers "=>4," Pipeline.batch.size "=>125," Pipeline.batch.delay "=>50} [2018-04-26t14 : 40:04,901][info][logstash.pipeline] Pipeline started succesfully {:p ipeline_id=> "main",:thread=> "#<thr Ead:0x518728c7 run> "} [2018-04-26t14:40:04,989][info][logstash.inputs.syslog] starting syslog UDP listener {: Addre Ss=> "0.0.0.0:514"} [2018-04-26t14:40:05,013][info][logstash.inputs.syslog] starting syslog TCP Listener {: Address = "0.0.0.0:514"} [2018-04-26t14:40:05,034][info][logstash.agent] Pipelines running {: count=>1,:p ipeline
 s=>["main"]}

View Port Numbers

[Root@node1 config]# netstat-anp|grep 514
tcp6       0      0::: 514                  :::*                    LISTEN      4260/java           
UDP        0      0 0.0.0.0:514             0.0.0.0:*                           4260/java           
Unix  2      [ACC]     STREAM     LISTENING     15141    822/mcelog           /var/run/mcelog-client
Unix  2      []         dgram                    15147    828/chronyd          
[Root@node1 config]#

found that port 514 has been Logstash 2, syslog terminal

Switch to another server Node2, configure syslog

[Root@node2 ~]# vi/etc/rsyslog.conf

Add a line of *. * @ @node1:514, and send the logs to the remote Logstash.

# remote host is:name/ip:port, e.g 192.168.0.1:514, port optional
#*.* @ @remote-host:514
* * @ @node1:514

Re-RSYSLOGD the program

[Root@node2 ~]# systemctl Restart Rsyslog

3. Collect data at logstash end

At this point, we find that the Logstash end has collected the Syslog log data of Node2

[2018-04-26t14:45:18,361] [INFO] [Logstash.inputs.syslog] New connection {:client=> "10.17.12.157:55204"} {"Severity_label" = "Informational"
              , "Facility_label" = "System", "timestamp" and "APR-14:39:23", "Severity" and "6", "Host" = "10.17.12.157", "message" = "Stopping System Logging service...\n", "@version "=" 1 "," program "=" Systemd "," @timestamp "and" 2018-04-26t06:39:23.000z "," Typ
    E "=" rsyslog "," priority "=" logsource "=" Node2 "," facility "= 3} { "Severity_label" = "informational", "Facility_label" and "system", "timestamp" = "APR 26 14:39:23" "," severity "= 6," host "=" 10.17.12.157 "," message "=" Stopped System Log " Ging service.\n "," @version "and" 1 "," program "=" Systemd "," @timestamp "and" = " 2018-04-26t06:39:23.000z, "type" = "Rsyslog", "priority", "Logsource" and "=" "Node2", "facility" = 3} {"Severity_label" = "Notice", "Facility_label" and "Security/autho" Rization "," timestamp "=" Apr 14:39:23 "," severity "+ 5," host "=" 10.17.1 " 2.157 "," message "=" Unregistered authentication Agent for unix-process:4601:59761164 (System bus name:1.2
          556, Object path/org/freedesktop/policykit1/authenticationagent, Locale En_us.utf8) (disconnected from bus) \ n ",
              "@version" and "1", "program" = "POLKITD", "@timestamp" and "2018-04-26t06:39:23.000z", "Type" = "Rsyslog", "priority" = +, "pid" = "762", "logsource" = "Node2", "facility" = "Severity_label", "informational", "Facility_label" and "system" , "timestamp" => "APR 14:40:01", "severity" + 6, "host" = "10.17.12.157", "message" =&gt ; "Started Session 1235 of user root.\n", "@version" and "1", "program" = "Systemd", "@ti Mestamp "= 2018-04-26t06:40:01.000z," type "=" Rsyslog "," Priority "," Lo Gsource "+" Node2 "," facility "= 3} {" Severity_label "=" informational "," Facility_label " = "System", "timestamp" = "Apr 14:40:01", "Severity" and "6", "Host" and "10"
           .17.12.157 "," message "=" Starting Session 1235 of user root.\n "," @version "and" 1 ",
          "Program" = "Systemd", "@timestamp" = 2018-04-26t06:40:01.000z, "type" = "Rsyslog", "Priority" = "logsource" = "Node2", "facility" = 3} {"Severity_label" =&G T
  "Informational",  "Facility_label" = "clock", "timestamp" and "APR-14:40:01", "Severity" and "6", "Host" = "10.17.12.157", "message" = "(root) CMD (/USR/LIB64/SA/SA1 1 1) \ n", "@version" =&G T "1", "program" = "CROND", "@timestamp" and "= 2018-04-26t06:40:01.000z", "type" and "="
          Rsyslog "," priority "= +," pid "=" 4640 "," Logsource "and" Node2 ",
         "Facility" = 9} {"Severity_label" = "Notice", "Facility_label" and "Security/authorization", "Timestamp" = "Apr 14:45:18", "Severity" and 5, "host" and "10.17.12.157", " Message "= =" Registered authentication Agent for unix-process:4786:59796608 (System bus name:1.2559 [/usr/bin/pkttyag
          ENT--notify-fd 5--fallback], object path/org/freedesktop/policykit1/authenticationagent, locale En_us.utf8) \ n ", "@version" = "1"Program" = "POLKITD", "@timestamp" and "= 2018-04-26t06:45:18.000z", "type" and "=" Rsyslog "," priority "=--" pid "=" 762 "," logsource "=" Node2 "," Facility "= ten} {" Severity_label "=" informational "," Facility_label "and" system "," Timestam P "=" Apr 14:45:18 "," severity "= 6," host "=" 10.17.12.157 "," message "
        = "Starting System Logging service...\n", "@version" and "1", "program" = "Systemd",
         "@timestamp" = 2018-04-26t06:45:18.000z, "type" = "Rsyslog", "priority" = 30, "Logsource" = "Node2", "facility" = 3} {"Severity_label" = "informational", "facility_l Abel "and" System "," timestamp "and" APR-14:45:18 "," Severity "and" 6 "," host "=&G T
     "10.17.12.157",      "Message" = "Started System Logging service.\n", "@version" and "1", "program" = "SYS" Temd "," @timestamp "= 2018-04-26t06:45:18.000z," type "=" Rsyslog "," priority "=&G T "Logsource" = "Node2", "facility" = 3} {"Severity_label" = "Notice", "Facilit
              Y_label "=" and "Security/authorization", "timestamp" and "APR-14:45:18", "Severity" and 5, "Host" = "10.17.12.157", "message" = "Unregistered authentication Agent for unix-process:4786: 59796608 (System bus name:1.2559, Object path/org/freedesktop/policykit1/authenticationagent, Locale En_us.utf8) ( Disconnected from bus) \ n "," @version "=" 1 "," program "=" POLKITD "," @timestamp "=&gt ; 2018-04-26t06:45:18.000z, "type" = "Rsyslog", "priority" =--"pid" = "762", "LOGSOURCE "=" Node2 "," facility "=" Severity_label "+" informational "," facility_label "+ = "Syslogd", "timestamp" and "Apr 14:45:18", "severity" = 6, "host" and "10.17". 12.157 "," message "=" [Origin software=\ "rsyslogd\" swversion=\ "7.4.7\" x-pid=\ "4792\" x-info=\ "http://www.  Rsyslog.com\ "] start\n", "@version" and "1", "program" = "Rsyslogd", "@timestamp" and "="  2018-04-26t06:45:18.000z, "type" = "Rsyslog", "priority", "Logsource", "=" "Node2", "facility" = 5}