Logstash TCP connection leak

[red888 (red der)]

Logstash: 5.3.0

Logstash is keeping TCP connections open even for servers that are shutdown. It also seems to be preventing the OS from closing the connections via TCP keep alive- I just have the default settings:

  # cat /proc/sys/net/ipv4/tcp_keepalive_time
  7200
  # cat /proc/sys/net/ipv4/tcp_keepalive_intvl
  75
  # cat /proc/sys/net/ipv4/tcp_keepalive_probes
  9

I also have duplicate tcp connections for the same hosts.

So I added `data_timeout => 500` to my input filter and restarted logstash but it looks like connections are still growing and remaining in the established state.

I think logstash is keeping the tcp connection active even if the server on the other end is shutdown? If the OS sees the connection as ESTABLISHED does that not count as an idle connection? Because in that case data_timeout would have no impact right?

I was hoping there would be a logstash setting for this, but it seems like logstash is even preventing the OS from cleaning up connections. I don't want to have to run a cronjob to restart logstash.

Edit: so it seems logstash doesn't implement TCP keep alives so my keep alive settings won't matter anyway and it will always leak TCP connections.

Should I open a bug report on github for this?

[jakelandis]

I assume this is in reference to the TCP input ?

If so,
We just released a re-implementation of the TCP-input that uses a NIO under the covers, for non-ssl connection. If this is non-ssl connection, I would encourage you to try the new implementation, and if it still has issues, then please raise an issue.

To upgrade the TCP input:

1. Edit the Gemfile under your Logstash home directory
2. Update to lock to the v4 of plugin ( add , "~> 4" to the existing line): gem "logstash-input-tcp", "~> 4"
3. run the command bin/logstash-plugin update logstash-input-tcp

This will be the version shipped with 5.6.0+

[red888 (red der)]

Yup this is TCP input and its non-ssl.

I'm planning an ELK upgrade soon so I'll wait for 5.6.0- although it looks like your next release is going to be 6.0.0?

For now scheduling reboots to clear out connections works fine for me.

[jakelandis]

There will be a 5.6.0 release about the same time as the 6.0.0-beta1 release [1]. The new tcp-input with nio will be in both.

[1] standard disclaimer - assuming plans don't change.