UDP listener died (SYSLOG) when running as a service

Elastic Stack Logstash
1

Hello and thanks for taking a moment of your time to read over this.

I am running Logstash as a collector for SYSLOG messages from networking equipment. Executing the binary while pointing to the configuration file with the -f flag works perfectly; it's only when I rely on the system's service (managed via systemd) do I find that Logstash fails to receive any data from my devices.

Logstash configuration file contents:

input {
  udp {
    type => "syslog"
    port => "514"
  }
}
output {
  elasticsearch {
    hosts => ["localhost:9200"]
    index => "my_index_name_%{+YYYYMMdd}"
  }
  stdout {
    codec => rubydebug
  }
  if [type] == "syslog" {
    kafka {
      codec => json
      topic_id => "junos_syslog"
    }
  }
}

host: RHEL 7
installed via: YUM package manager
selinux: disabled
configuration file location / permissions:

➜  ~ ls -l /etc/logstash/conf.d/             
total 4
-rw-r--r-- 1 logstash logstash 1613 Apr 19 18:41 logstash.conf

journalctl logs:

➜  ~ sudo journalctl -u logstash -f
Password: 
-- Logs begin at Tue 2018-04-17 08:13:17 CDT. --
Apr 19 18:06:03 hounett01 systemd[1]: Stopping logstash...
Apr 19 18:06:04 hounett01 systemd[1]: Started logstash.
Apr 19 18:06:04 hounett01 systemd[1]: Starting logstash...
Apr 19 18:06:19 hounett01 logstash[32414]: Sending Logstash's logs to /var/log/logstash which is now configured via log4j2.properties
Apr 19 18:14:21 hounett01 systemd[1]: Stopping logstash...
Apr 19 18:14:22 hounett01 systemd[1]: Started logstash.
Apr 19 18:14:22 hounett01 systemd[1]: Starting logstash...
Apr 19 18:14:37 hounett01 logstash[1087]: Sending Logstash's logs to /var/log/logstash which is now configured via log4j2.properties
Apr 19 18:18:50 hounett01 systemd[1]: Stopping logstash...
Apr 19 18:18:52 hounett01 systemd[1]: Stopped logstash.

I have increased the debugging level within my logstash.yml file and found that my system is quite upset with the binding to the well-known SYSLOG port of 514

[2018-04-20T09:00:34,036][INFO ][logstash.inputs.udp      ] Starting UDP listener {:address=>"0.0.0.0:514"}
[2018-04-20T09:00:34,039][WARN ][logstash.inputs.udp      ] UDP listener died {:exception=>#<Errno::EACCES: Permission denied - bind(2) for "0.0.0.0" port 514>, :backtrace=>["org/jruby/ext/socket/RubyUDPSocket.java:197:in `bind'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-input-udp-3.3.1/lib/logstash/inputs/udp.rb:101:in `udp_listener'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-input-udp-3.3.1/lib/logstash/inputs/udp.rb:57:in `run'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:514:in `inputworker'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:507:in `block in start_input'"]}

I'm curious as to understand how we can leverage Logstash to run as a syslog collector if the service cannot request the proper escalation of privileges to run as a service? More importantly, is there anything we can do to correct this issue?

2

514 is a privileged port so you can't bind to it unless you're running Logstash as the root user. I generally just set a different port outside of the privileged port range (like 5001, for instance).

Change your syslog port and your listener should be able to bind and startup without problems.

3

Understood, the challenge that I have is that the network devices are typically dumb and do not allow for custom ports to be sent out.

There are other network services that are operating on well-known ports that we are trying to ingest (SNMP traps, DHCP Requests, etc.), those of which are also very difficult (if not impossible) to change.

Is there a way for the logstash service to request the appropriate permissions needed when running as a service?

4

For instance, if you installed Apache or NGINX, systemd would be able to bind to the well-known ports of TCP 80 and 443. for some reason, Logstash isn't able to bind to the well known port of 514

5

I don't have the information in front of me, so this is more of a hint so that you can maybe search for it online. There are two options...

  1. Configure Logstash to listen on a higher port number (e.g. 55514), and use iptables to direct traffic received on 514 to 55514.

  2. Configure Java to be able to open network connection with elevated privileges.

Sorry I couldn't give you exact instructions, but I hope that points you in the right direction.

6

This worked out well for us, thank you for the suggestion!