Logstash tcp multihost output (multi-target host output to ensure the stability of the TCP output link), logstashmultihost

Logstash tcp multihost output (multi-target host output to ensure the stability of the TCP output link), logstashmultihost

When cleaning logs, there is an application scenario, that is, when the TCP output, you need to switch to the next available entry when a host fails, the original tcp output only supports setting a single target host. Therefore, I developed the tcp_multihost output plug-in based on the original tcp to meet this scenario.

The plug-in selects a random link at the beginning, and tries the next host in the array after a link error exceeds 3 (default) consecutive times.

Github: http://github.com/xiaohelong2005

Logstash version: 1.4.2

File Location:

# encoding: utf-8require "logstash/outputs/base"require "logstash/namespace"require "thread"#@auhor:xiaohelong #@date:2014-10-24#@email:xiaohelong2005@gmail.com# Write events over a TCP socket.#Auto select the host from iplist to tolerate the link# Each event json is separated by a newline.## Can  connect to a server,class LogStash::Outputs::Tcp_Multihost < LogStash::Outputs::Base  config_name "tcp_multihost"  milestone  0  default :codec, "json"  # the address to connect to.  #hosts config example  config :hosts, :validate => :array, :required => true,:default=>{}    config :reconnect_times, :validate => :number,:default=>3  # When connect failed,retry interval in sec.  config :reconnect_interval, :validate => :number, :default => 10 #last available host @hosts_size=0 #last available host ip@host="0.0.0.0"#last available port@port=9200#retry action count,if retry_count<=0,we need to update the host and port , also include retry_count itself@retry_count=0#get the desgined index data@initloc=0  public  def register    require "stud/try"  #here we use the recorded host,if recorded host is not available, we need update it  @retry_count=@reconnect_times   @hosts_size=@hosts.length    @logger.info("length:#@hosts_size; hosts:#@hosts")   @initloc=Random.rand(@hosts_size)#generate 0-(hosts_size-1) int    @logger.info("initloc:#@initloc")    icount=0;      @hosts.each do |hosthash|                           @logger.info("hosthash info",hosthash)       end#do        @host=@hosts[@initloc].keys[0]               @port=@hosts[@initloc][@host]             client_socket = nil      @codec.on_event do |payload|        begin            @retry_count=@reconnect_times#here we need to init retry mark          client_socket = connect unless client_socket          r,w,e = IO.select([client_socket], [client_socket], [client_socket], nil)                    # don't expect any reads, but a readable socket might          # mean the remote end closed, so read it and throw it away.          # we'll get an EOFError if it happens.          client_socket.sysread(16384) if r.any?          # Now send the payload          client_socket.syswrite(payload) if w.any?           @logger.info("tcp output info:", :host => @host, :port => @port,                       :exception => e, :backtrace => e.backtrace)        rescue => e          @logger.warn("tcp output exception", :host => @host, :port => @port,                       :exception => e, :backtrace => e.backtrace)          client_socket.close rescue nil          client_socket = nil          @retry_count-=1            @logger.info("retry_count:#@retry_count")          if    @retry_count<=0                                        @initloc+=1                              @initloc=@initloc%@hosts_size #update  init location                                           @host=@hosts[@initloc].keys[0]                                                    @port=@hosts[@initloc][@host]                   @retry_count=@reconnect_times #update retry_count                      @logger.info("retry_count <=0,initloc:#@initloc,retry_count=#@retry_count:", :host => @host, :port => @port, :exception => e, :backtrace => e.backtrace)          end          sleep @reconnect_interval          retry        end      end  end # def register  private  def connect    Stud::try do      return TCPSocket.new(@host,@port)    end  end # def connect  public  def receive(event)    return unless output?(event)    @codec.encode(event)  end # def receiveend # class LogStash::Outputs::Tcp_multihost

Configuration instructions (I put it in LOGSTASH_HOME/config ):

Output {

Tcp_multihost {

Hosts => [
{& Quot; 127.0.0.1 & quot; = & quot; 9202 & quot "},
{& Quot; localhost & quot; = & quot; 9201 & quot "},
{& Quot; 127.0.0.1 & quot; = & quot; 9203 & quot "},
{"127.0.0.1" => "9204 "}
] # Host list
Workers => 16 # thread, default value: 1

Reconnect_times => 3 # The default value is 3. Switch the number of attempts

Reconnect_interval => 3 # The default value is 10 seconds. The reconnection interval fails.

}

}

Call execution:

"LOGSTASH_HOME/bin/logstash" agent -- debug-f "LOGSTASH_HOME/config/shipper. config" -- pluginpath "LOGSTASH_HOME"

NC acceptor can try:

Nc-lkv 9201