Logstash tcp multihost output (multi-target host output to ensure the stability of the TCP output link), logstashmultihost
When cleaning logs, there is an application scenario, that is, when the TCP output, you need to switch to the next available entry when a host fails, the original tcp output only supports setting a single target host. Therefore, I developed the tcp_multihost output plug-in based on the original tcp to meet this scenario.
The plug-in selects a random link at the beginning, and tries the next host in the array after a link error exceeds 3 (default) consecutive times.
Github: http://github.com/xiaohelong2005
Logstash version: 1.4.2
File Location:
# encoding: utf-8require "logstash/outputs/base"require "logstash/namespace"require "thread"#@auhor:xiaohelong #@date:2014-10-24#@email:xiaohelong2005@gmail.com# Write events over a TCP socket.#Auto select the host from iplist to tolerate the link# Each event json is separated by a newline.## Can connect to a server,class LogStash::Outputs::Tcp_Multihost < LogStash::Outputs::Base config_name "tcp_multihost" milestone 0 default :codec, "json" # the address to connect to. #hosts config example config :hosts, :validate => :array, :required => true,:default=>{} config :reconnect_times, :validate => :number,:default=>3 # When connect failed,retry interval in sec. config :reconnect_interval, :validate => :number, :default => 10 #last available host @hosts_size=0 #last available host ip@host="0.0.0.0"#last available port@port=9200#retry action count,if retry_count<=0,we need to update the host and port , also include retry_count itself@retry_count=0#get the desgined index data@initloc=0 public def register require "stud/try" #here we use the recorded host,if recorded host is not available, we need update it @retry_count=@reconnect_times @hosts_size=@hosts.length @logger.info("length:#@hosts_size; hosts:#@hosts") @initloc=Random.rand(@hosts_size)#generate 0-(hosts_size-1) int @logger.info("initloc:#@initloc") icount=0; @hosts.each do |hosthash| @logger.info("hosthash info",hosthash) end#do @host=@hosts[@initloc].keys[0] @port=@hosts[@initloc][@host] client_socket = nil @codec.on_event do |payload| begin @retry_count=@reconnect_times#here we need to init retry mark client_socket = connect unless client_socket r,w,e = IO.select([client_socket], [client_socket], [client_socket], nil) # don't expect any reads, but a readable socket might # mean the remote end closed, so read it and throw it away. # we'll get an EOFError if it happens. client_socket.sysread(16384) if r.any? # Now send the payload client_socket.syswrite(payload) if w.any? @logger.info("tcp output info:", :host => @host, :port => @port, :exception => e, :backtrace => e.backtrace) rescue => e @logger.warn("tcp output exception", :host => @host, :port => @port, :exception => e, :backtrace => e.backtrace) client_socket.close rescue nil client_socket = nil @retry_count-=1 @logger.info("retry_count:#@retry_count") if @retry_count<=0 @initloc+=1 @initloc=@initloc%@hosts_size #update init location @host=@hosts[@initloc].keys[0] @port=@hosts[@initloc][@host] @retry_count=@reconnect_times #update retry_count @logger.info("retry_count <=0,initloc:#@initloc,retry_count=#@retry_count:", :host => @host, :port => @port, :exception => e, :backtrace => e.backtrace) end sleep @reconnect_interval retry end end end # def register private def connect Stud::try do return TCPSocket.new(@host,@port) end end # def connect public def receive(event) return unless output?(event) @codec.encode(event) end # def receiveend # class LogStash::Outputs::Tcp_multihost
Configuration instructions (I put it in LOGSTASH_HOME/config ):
Output {
Tcp_multihost {
Hosts => [
{& Quot; 127.0.0.1 & quot; = & quot; 9202 & quot "},
{& Quot; localhost & quot; = & quot; 9201 & quot "},
{& Quot; 127.0.0.1 & quot; = & quot; 9203 & quot "},
{"127.0.0.1" => "9204 "}
] # Host list
Workers => 16 # thread, default value: 1
Reconnect_times => 3 # The default value is 3. Switch the number of attempts
Reconnect_interval => 3 # The default value is 10 seconds. The reconnection interval fails.
}
}
Call execution:
"LOGSTASH_HOME/bin/logstash" agent -- debug-f "LOGSTASH_HOME/config/shipper. config" -- pluginpath "LOGSTASH_HOME"
NC acceptor can try:
Nc-lkv 9201