Syslog input always dying

Elastic Stack Logstash
1

Hi there!

We configured multiple syslog inputs for different incoming log types (switches, firewalls, etc), each of them with a dedicated port, like this:

input {
  syslog {
    host => "0.0.0.0"
    type => "switches"
    port => 5519
  }
}

Since I restarted logstash with this configuration, it always crashes, sometimes after five minutes, sometimes after half an hour, but the crashes are inevitable.

Log:

{:timestamp=>"2015-06-15T11:54:24.267000+0200", :message=>"syslog listener died", :protocol=>:udp, :address=>"0.0.0.0:5519", :exception=>#<SocketError: recvfrom: name or service not known>, :backtrace=>["/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-input-syslog-0.1.6/lib/logstash/inputs/syslog.rb:138:in `udp_listener'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-input-syslog-0.1.6/lib/logstash/inputs/syslog.rb:117:in `server'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-input-syslog-0.1.6/lib/logstash/inputs/syslog.rb:97:in `run'"], :level=>:warn}

This is really annoying and I don't know what to do with this... are multiple syslog inputs a problem for logstash? :frowning:
Thanks in advance!

2

What version are you on?

3

Logstash: 1.5
Elasticsearch: 1.6

I'm using the versions from the debian repository btw.

EDIT: Update. I replaced all my simultanous syslog inputs with "tcp" inputs, but to no avail:

{:timestamp=>"2015-06-18T09:48:47.541000+0200", :message=>"An error occurred. Closing connection", :client=>"172.27.10.131:39532", :exception=>#, :backtrace=>["org/jruby/RubyIO.java:3018:in sysread'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-input-tcp-0.1.5/lib/logstash/inputs/tcp.rb:164:inread'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-input-tcp-0.1.5/lib/logstash/inputs/tcp.rb:112:in handle_socket'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-input-tcp-0.1.5/lib/logstash/inputs/tcp.rb:147:inclient_thread'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-input-tcp-0.1.5/lib/logstash/inputs/tcp.rb:145:in `client_thread'"], :level=>:error}

I also reported this behaviour here: GitHub issue

4

Okay, maybe this helps:

I changed a line in the syslog input gem file (/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-input-syslog-0.1.6/lib/logstash/inputs/syslog.rb in my case). There's a part, where the TCP listener is defined:

def tcp_listener(output_queue)
@logger.info("Starting syslog tcp listener", :address => "#{@host}:#{@port}")
@tcp = TCPServer.new(@host, @port)

loop do
  socket = @tcp.accept
  @tcp_sockets << socket

  break if @shutdown_requested.true?

  Thread.new(output_queue, socket) do |output_queue, socket|
    tcp_receiver(output_queue, socket)
  end 
end

ensure
close_tcp
end # def tcp_listener

I replaced "
break if @shutdown_requested.true?"
with

" break if @shutdown_requested.false?" and logstash hasn't crashed since then. This was three days ago, normally logstash would crash after a few hours at the latest. It is a dirty hack, but it seems to work...