No input via udp/syslog plugin

Elastic Stack Logstash
1

My logstash-1.5.4 install does not collect any logs via the udp or the syslog input plugin anymore. I have the following config file:

input {
   file{
       path => "/opt/logs/tcpdump.log"
   }
  udp {
    port => "514"
    type => "syslog"
  }
   syslog{
     type => "syslog"
   }
}

Whenever I quote out udp and syslog logstash runs normal and it collects logs via the file input command and sends it to my output but enabling udp or syslog generates no logs at all. I have checked port 514 with tcpdump and see that input does arrive there. Also when I close logstash if I have udp or tcp enabled it doesn't shut down properly and I have to kill the running process before being able to restart logstash.

Nothing appears in the logstash log if I use the -l option. I have tried updating both syslog and udp plugin but the same thing happens.

2

Are you running Logstash as root? Is there anything in the Logstash log if you increase the logging verbosity with --verbose or --debug?

3

I am running as root and with verbose I am getting the following logging:

{:timestamp=>"2015-09-22T07:57:11.951000-0400", :message=>"Pipeline started", :level=>:info}
{:timestamp=>"2015-09-22T07:57:15.973000-0400", :message=>"Starting syslog udp listener", :address=>"0.0.0.0:514", :level=>:info}
{:timestamp=>"2015-09-22T07:57:15.974000-0400", :message=>"syslog listener died", :protocol=>:udp, :address=>"0.0.0.0:514", :exception=>#<SocketError: bind: name or service not known>, :backtrace=>["org/jruby/ext/socket/RubyUDPSocket.java:160:in `bind'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-input-syslog-1.0.1/lib/logstash/inputs/syslog.rb:135:in `udp_listener'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-input-syslog-1.0.1/lib/logstash/inputs/syslog.rb:117:in `server'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-input-syslog-1.0.1/lib/logstash/inputs/syslog.rb:97:in `run'"], :level=>:warn}
{:timestamp=>"2015-09-22T07:57:15.995000-0400", :message=>"Starting syslog tcp listener", :address=>"0.0.0.0:514", :level=>:info}
{:timestamp=>"2015-09-22T07:57:15.996000-0400", :message=>"syslog listener died", :protocol=>:tcp, :address=>"0.0.0.0:514", :exception=>#<Errno::EADDRINUSE: Address already in use - bind - Address already in use>, :backtrace=>["org/jruby/ext/socket/RubyTCPServer.java:118:in `initialize'", "org/jruby/RubyIO.java:853:in `new'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-input-syslog-1.0.1/lib/logstash/inputs/syslog.rb:152:in `tcp_listener'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-input-syslog-1.0.1/lib/logstash/inputs/syslog.rb:117:in `server'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-input-syslog-1.0.1/lib/logstash/inputs/syslog.rb:101:in `run'"], :level=>:warn}
{:timestamp=>"2015-09-22T07:57:20.975000-0400", :message=>"Starting syslog udp listener", :address=>"0.0.0.0:514", :level=>:info}
4

Oh, right. The syslog input defaults to port 514 so you can't have a UDP listener on the same port. Pick one. You don't have any other software (like rsyslog) that listens on port 514? Check with lsof?

5

I don't have both active at the same time, I used to run it with the udp-input but when that stopped working I tried it with the syslog-input instead. The logging I just posted was from only having the syslog-input active, udp was commented out.

If I shut down logstash and check with netstat there is nothing else running on port 514.

6

I just tried out another port for the udp input and on the first run it works great, but after I close logstash and try to reboot it that secondary port also fails. So it has to be some sort of leftover process somewhere, although both lsof and netstat report nothing left running on the port.