Log monitoring _elasticstack5-0001.logstash fast normalized deployment and background operation?

Technology Stack Line:

650) this.width=650; "src=" Http://s3.51cto.com/wyfs02/M02/8B/42/wKiom1hIDFyjbuIAAAJMIsTSBRQ088.png "title=" Clipboard.png "alt=" Wkiom1hidfyjbuiaaajmistsbrq088.png "/>

1. Support flexible processing mode, Elasticsearch is real-time full-text index, do not need to like other product sample pre-programmed to achieve

2. Support simple configuration, Elasticsearch is all using JSON interface, Logstash is a ruby DSL design, are common configuration syntax

3. Support Efficient data retrieval, although every query is real-time calculation, but basically can realize the second-level response of all-day data query

4. Support Convenient linear expansion, whether it is Elasticsearch cluster or Logstash cluster can be linearly extended

4. Support Brilliant front-end display, Kibana interface only click on the mouse, you can complete the search, aggregation, production of brilliant dashboards

General Architecture:

650) this.width=650; "src=" Http://s5.51cto.com/wyfs02/M01/8B/42/wKiom1hIDRzBOyZTAAEVdAxE0LY758.png "title=" Limanman.png "alt=" Wkiom1hidrzboyztaaevdaxe0ly758.png "/>

Brief introduction:

Description: Logstash as a receiving, processing, output log tool, can handle a variety of logs, written by JRuby, supporting a large number of ancillary plug-ins, in 2013 was Elasticsearch acquired, formally become a member of elastic stack.

Current Package Installation:

Rpm-qa |grep JDK |xargs rpm-e--nodepsyum-y install gcc cmake coreutils lsof pcre pcre-devel zlib zlib-devel coreutils Lrzsz wget python-setuptools epel-releaseyum install java-1.8.0-openjdk java-1.8.0-openjdk-devel NC jqrpm-ivh https://a rtifacts.elastic.co/downloads/logstash/logstash-5.0.0.rpm

SOURCE Deployment:

# Install JAVA runtime Environment Rpm-qa |grep JDK |xargs rpm-e--nodepsyum-y install gcc cmake coreutils lsof pcre pcre-devel zlib zlib-d Evel coreutils lrzsz wget python-setuptools epel-releaseyum install JAVA-1.8.0-OPENJDK java-1.8.0-openjdk-devel NC jqjav a-version# Create logstash directory mkdir-p/xm-workspace/xm-apps/logstash/tar-zxvf logstash-5.0.0.tar.gzmv logstash-5.0.0/*/ xm-workspace/xm-apps/logstash/# creating a profile directory Mkdir-p/xm-workspace/xm-apps/logstash/etc/logstash.d

Common parameters:

-N	Specifies the name of the Logstash instance and, if not specified, the host name by default
-F	Specifies the Logstash configuration file or directory, and if the directory is specified, merge the configuration files in the directory
-E	Specifies logstash configuration data, specified as a string, often used for manual testing
-L	Specify Logstash log file directory, logs directory under default installation directory
--debug	Turn on Logstash debug mode
--log.level	Specify the Logstash log level, Fatal/error/warn/info/debug/trace
--config.debug	Specify logstash Print configuration merge log, open if debug mode--debug and set log level
-T	Test the Logstash configuration file, which must be used with-f
-R	Set Logstash Auto Reload profile, must be used with-f
--config.reload.interval	Set Logstash overload configuration cycle, default to 3 seconds
--http.host	Setting the LOGSTASHAPI Listener address
--http.port	Setting the Logstashapi listening port
--quiet	Set Logstash to Quiet mode

Note: When the-f directory is specified, Logstash automatically reads all the *.conf configuration files in the directory, sorts them alphabetically, and then stitching them into a full large configuration file in memory, and recommends naming the profile with a digital numbering method. At the same time in the configuration of rigorous use if to determine the different log action, there is 5.0 start all command line configuration can be set through the/xm-workspace/xm-apps/logstash/config/logstash.yml file.

Quick test:

/xm-workspace/xm-apps/logstash/bin/logstash-e ' Input{stdin{}}output{stdout{codec=>rubydebug}} '--debug

{"@timestamp" = 2016-12-06t08:37:52.922z, "@version" = "1", "host" = "Yunchanpinxian", "Message" = "Limanman"}

Note: Data is passed between threads in the form of an event object, Logstash automatically adds some extra information to the event object, and most importantly @timestamp, which marks the occurrence of the event, which flows internally in the form of a Joda time object. It also supports adding a delete tag array and a custom field to the event object through Add_tag/remove_tag/add_field/remove_field in the filter plug-in.

Configuration syntax:

1. Logstash designed its own DSL (similar to the puppet DSL) to describe the configuration, using the {} definition zone, which can contain multiple plug-in areas within the plug-in domain to define the corresponding region's key-value pair settings

2. Logstash supports a small number of data types, such as bool type (debug = True), String type (host = "hostname"), type # (port = 514), array type (match = > ["datetime", "UNIX", "ISO8601"]), hash type (options = Key1 = "Value1", Key2 = "value2"})

3. Logstash supports field references, because fields can be hash or array types, field objects can be restored through [field], and also support the right-reverse subscript to retrieve child elements, if you want to use a field reference in a string to pass%{[field][...]} Realize

4. Logstash support = = equals/!= not equal to/< less than/> greater than/<= is equal to/>= greater than or equal to/=~ match/!~ mismatch/in contains/not in does not contain/and with/or or/nand non-with/xor non-or/() Compound expression/! () take repeated expressions

If "_grokparsefailure" not in [tags] {} else if [status]!~/^2\d\d/or ([url] = = "/noc.gif" NAND [geoip][city]! = "Beiji Ng ") {} else {}

Running in the background:

#!/bin/sh# -*- coding: utf-8 -*-### authors: limanman# oschina: http:/ /xmdevops.blog.51cto.com/# purpose: control logstash start|stop|force-stop|status|restart  ### customer envname=logstashpidfile= "/var/run/${name}.pid" ls_home=/xm-workspace/xm-apps/ Logstashexport path=/sbin:/usr/sbin:/bin:/usr/bin:${ls_home}/bin# must use root if  [  ' id -u '  -ne 0 ]; then   echo  you need  Root privileges to run this script "   exit 1fi#  Optimizationsls_heap_size= "1024m" ls_open_files=102400# logstash comm# ls_opts= "--debug" LS_OPTS= "--quiet" ls_log_dir=${ls_home}/logsls_conf_dir= "${ls_home}/etc/logstash.d" [ ! -d ${ls_home}  ] && mkdir -p ${LS_HOME}[ ! -d ${LS_LOG_DIR} ]  && mkdir -p ${ls_loG_dir}[ ! -d ${ls_conf_dir} ] && mkdir -p ${ls_conf_dir} program=${ls_home}/bin/${name}args= "-f ${ls_conf_dir} -l ${ls_log_dir} ${ls_opts}" Start ()  {  ls_java_opts= "${ls_java_opts} -djava.io.tmpdir=${ls_home}"   HOME=${LS_HOME}   export PATH HOME LS_HEAP_SIZE LS_JAVA_OPTS LS_USE_GC_LOGGING   Ulimit -n ${ls_open_files}  # run the program!  bash -c   "    cd  $LS _home    ulimit -n ${ls_open_files}     exec \ "$program \"   $args    " 2> " ${ls_log_dir}/${name}- Error.log " &>/dev/null &  echo $! >  $pidfile   echo   "${name} started."   return 0}stop ()  {  if status ; then    pid= ' cat  ' $pidfile '     echo  ' killing ${name}  (pid  $pid)  with  sigterm "    kill -TERM  $pid     for i in  1 2 3 4 5 ; do      echo  "waiting  ${name}  (pid  $pid)  to die ... "      status | |  break      sleep 1    done     if status ; then      echo  "${name} stop  Failed; still running. "     else      echo  "${name} stopped."     fi  fi}status ()  {  if [ -f  "$pidfile"  ]  ; then    pid= ' cat  "$pidfile" '     if kill -0   $pid  > /dev/null 2> /dev/null ; then      return 0     else      return 2    fi   else    return 3  fi}force_stop ()  {  if status  ; then    stop    status && kill - kill  ' cat  "$pidfile"   fi}case  "$"  in  start)      status    code=$?    if [  $code  -eq 0 ];  then      echo  "${name} is already running"      else      start      code=$?     fi    exit  $code     ;;   stop)  stop ;;   forCe-stop)  force_stop ;;   status)      status    code=$?     if [  $code  -eq 0 ] ; then      echo  "${ Name} is running "    else      echo " ${name}  is not running "    fi    exit  $code      ;;   restart)          stop && start      ;;   *)     echo  "usage: ${scriptname} {start|stop|force-stop|status| Restart} " >&2    exit 3  ;; Esacexit $?

Note: In order for the Logstash to run in the background, you should never set the input {} section of the configuration file in the configuration directory to the stdin {} used for testing, whether Nohub ... & or screen or Supervisord management process. Otherwise, the process will automatically exit when it runs for a while.

This article is from the "Li-Yun Development Road" blog, please be sure to keep this source http://xmdevops.blog.51cto.com/11144840/1880496

Log monitoring _elasticstack5-0001.logstash fast normalized deployment and background operation?