Logstash tcp multihost output (multi-target host output to ensure the stability of the TCP output link), logstashmultihost

Source: Internet
Author: User
Tags logstash

Logstash tcp multihost output (multi-target host output to ensure the stability of the TCP output link), logstashmultihost

When cleaning logs, there is an application scenario, that is, when the TCP output, you need to switch to the next available entry when a host fails, the original tcp output only supports setting a single target host. Therefore, I developed the tcp_multihost output plug-in based on the original tcp to meet this scenario.

The plug-in selects a random link at the beginning, and tries the next host in the array after a link error exceeds 3 (default) consecutive times.

Github: http://github.com/xiaohelong2005

Logstash version: 1.4.2

File Location:

# encoding: utf-8require "logstash/outputs/base"require "logstash/namespace"require "thread"#@auhor:xiaohelong #@date:2014-10-24#@email:xiaohelong2005@gmail.com# Write events over a TCP socket.#Auto select the host from iplist to tolerate the link# Each event json is separated by a newline.## Can  connect to a server,class LogStash::Outputs::Tcp_Multihost < LogStash::Outputs::Base  config_name "tcp_multihost"  milestone  0  default :codec, "json"  # the address to connect to.  #hosts config example  config :hosts, :validate => :array, :required => true,:default=>{}    config :reconnect_times, :validate => :number,:default=>3  # When connect failed,retry interval in sec.  config :reconnect_interval, :validate => :number, :default => 10 #last available host @hosts_size=0 #last available host ip@host="0.0.0.0"#last available port@port=9200#retry action count,if retry_count<=0,we need to update the host and port , also include retry_count itself@retry_count=0#get the desgined index data@initloc=0  public  def register    require "stud/try"  #here we use the recorded host,if recorded host is not available, we need update it  @retry_count=@reconnect_times   @hosts_size=@hosts.length    @logger.info("length:#@hosts_size; hosts:#@hosts")   @initloc=Random.rand(@hosts_size)#generate 0-(hosts_size-1) int    @logger.info("initloc:#@initloc")    icount=0;      @hosts.each do |hosthash|                           @logger.info("hosthash info",hosthash)       end#do        @host=@hosts[@initloc].keys[0]               @port=@hosts[@initloc][@host]             client_socket = nil      @codec.on_event do |payload|        begin            @retry_count=@reconnect_times#here we need to init retry mark          client_socket = connect unless client_socket          r,w,e = IO.select([client_socket], [client_socket], [client_socket], nil)                    # don't expect any reads, but a readable socket might          # mean the remote end closed, so read it and throw it away.          # we'll get an EOFError if it happens.          client_socket.sysread(16384) if r.any?          # Now send the payload          client_socket.syswrite(payload) if w.any?           @logger.info("tcp output info:", :host => @host, :port => @port,                       :exception => e, :backtrace => e.backtrace)        rescue => e          @logger.warn("tcp output exception", :host => @host, :port => @port,                       :exception => e, :backtrace => e.backtrace)          client_socket.close rescue nil          client_socket = nil          @retry_count-=1            @logger.info("retry_count:#@retry_count")          if    @retry_count<=0                                        @initloc+=1                              @initloc=@initloc%@hosts_size #update  init location                                           @host=@hosts[@initloc].keys[0]                                                    @port=@hosts[@initloc][@host]                   @retry_count=@reconnect_times #update retry_count                      @logger.info("retry_count <=0,initloc:#@initloc,retry_count=#@retry_count:", :host => @host, :port => @port, :exception => e, :backtrace => e.backtrace)          end          sleep @reconnect_interval          retry        end      end  end # def register  private  def connect    Stud::try do      return TCPSocket.new(@host,@port)    end  end # def connect  public  def receive(event)    return unless output?(event)    @codec.encode(event)  end # def receiveend # class LogStash::Outputs::Tcp_multihost


Configuration instructions (I put it in LOGSTASH_HOME/config ):

Output {

Tcp_multihost {

Hosts => [
{& Quot; 127.0.0.1 & quot; = & quot; 9202 & quot "},
{& Quot; localhost & quot; = & quot; 9201 & quot "},
{& Quot; 127.0.0.1 & quot; = & quot; 9203 & quot "},
{"127.0.0.1" => "9204 "}
] # Host list
Workers => 16 # thread, default value: 1

Reconnect_times => 3 # The default value is 3. Switch the number of attempts
Reconnect_interval => 3 # The default value is 10 seconds. The reconnection interval fails.
}
}

Call execution: 

"LOGSTASH_HOME/bin/logstash" agent -- debug-f "LOGSTASH_HOME/config/shipper. config" -- pluginpath "LOGSTASH_HOME"

NC acceptor can try:

Nc-lkv 9201




Contact Us

The content source of this page is from Internet, which doesn't represent Alibaba Cloud's opinion; products and services mentioned on that page don't have any relationship with Alibaba Cloud. If the content of the page makes you feel confusing, please write us an email, we will handle the problem within 5 days after receiving your email.

If you find any instances of plagiarism from the community, please send an email to: info-contact@alibabacloud.com and provide relevant evidence. A staff member will contact you within 5 working days.

A Free Trial That Lets You Build Big!

Start building with 50+ products and up to 12 months usage for Elastic Compute Service

  • Sales Support

    1 on 1 presale consultation

  • After-Sales Support

    24/7 Technical Support 6 Free Tickets per Quarter Faster Response

  • Alibaba Cloud offers highly flexible support services tailored to meet your exact needs.