UDP Listener died, address already in use

Elastic Stack Logstash
1

Hi,

I have two pipeline configurations located in /etc/logstash/conf.d. I can make work both seperately by running /usr/share/logstash/bin/logstash -f /etc/logstash/conf.d/**.conf -r, and it sends data to elasticsearch as I want. When I try to run both together (with pipeline.yml), I get an error.

The input of the problematic pipeline esxi.conf is :

input{
  udp {
    host => "myip"
    port => "514"
  }
}

Here's my pipeline.yml :

- pipeline.id: switch
  path.config: "/etc/logstash/conf.d/switch.conf"
- pipeline.id: esxi
  path.config: "/etc/logstash/conf.d/esxi.conf"

And here's the error I get when I try to run logstash (with systemctl start logstash) :

sept. 13 12:30:25 local logstash[4303]: [2022-09-13T12:30:25,350][ERROR][logstash.inputs.udp]
[esxi-pipeline][5beb5edfc5ff315c8ba13ccf64d80fa654d6b00786d512a26d6639b7cbfeea2f] UDP listener died 
{:exception=>#<Errno::EADDRINUSE: Address already in use - bind - Permission non accordée
bind(2) for "myip" port 514>, :backtrace=>["org/jruby/ext/socket/RubyUDPSocket.java:200:in `bind'", 
"/usr/share/logstash/vendor/bundle/jruby/2.6.0/gems/logstash-input-udp-3.5.0/lib/logstash/inputs/udp.rb:129:in `udp_listener'", 
"/usr/share/logstash/vendor/bundle/jruby/2.6.0/gems/logstash-input-udp-3.5.0/lib/logstash/inputs/udp.rb:81:in `run'", 
"/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:410:in `inputworker'", 
"/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:401:in `block in start_input'"]}

The switch logs are correctly ingested so I don't think that the problem is Logstash or my pipeline config.
What I don't understand is that the 514 port is not in use. Rsyslog is disabled and with netstat -tulpn, there's no mention of 514. And by running alone esxi.conf, it's working as supposed to work.

I rebooted ; I tried to allow ports below 1024 to non-root users with setcap cap_net_bind_service /usr/share/logstash/bin/logstash and with other file paths but it just broke everything. (Same problem as here : logstash will not open a listening port. · Issue #1587 · elastic/logstash · GitHub)

Is there a way to make it work ?

Thanks for your help !

2

Try to start logstash as root user, from the command line, just for testing purpose.

3

You also have this in your error message:

Permission non accordée

The port 514 may not be in use, just the logstash user does not have permission to even check it.

This also does not work

setcap cap_net_bind_service /usr/share/logstash/bin/logstash

Logstash is a Java processes, so you need to use setcap with the java binary of Logstash.

Check this old blog post on how to use the privileged port with logstash.

I would not recommend listening on port 514, if you really need to use this port a better option would be to use your Rsyslog to redirect the logs to a higher port on Logstash in the same server.

4

Hi @leandrojmp,

I admit that I used the command setcap without really knowing what I was doing. I also tried with /usr/share/logstash/jdk/bin/java but I wasn't aware of the symlink thing to do to resolve the libjli.so error explained in the link. Thanks for the link, I hadn't found it while searching.
Yes I cannot use something else than 514.

I had found a command using iptables : iptables -t nat -A PREROUTING -i ens160 -p udp -m udp --dport 514 -j REDIRECT --to-port 5147.

Do you recommand using the command above + modifying my esxi.conf with 5147 ?
Or enable syslog as you said and modify rsyslog.conf ? I'm pretty new to this, I don't know where I would have to specify something else in rsyslog.

Thank you so much !

5

EDIT

I ran iptables -t nat -A PREROUTING -i ens160 -p udp -m udp --dport 514 -j REDIRECT --to-port 5147 and it is working now.

6

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.