Logstash CLOSE_WAIT and Filbe beat I/0 timeout

oraclept · February 6, 2018, 1:15pm

In Filebeat logs I am seeing Filebeat I/O timeout with logstash tcp connection going infinite CLOSE_WAIT. until logstash stops processing. Dont see anything in logstashlogs.

Logstash and filebeat both are 5.4 version. Not sure what is causing the issue. It was all fine until yesterday.

Logstash completely stops processing log messages.

Is there any setting in Logstash beats inputs to close CLOSE_WAIT connections ?

oraclept · February 6, 2018, 1:21pm

it worked well for year. I did not change any logstash and filebeat configuration. Not sure why I am hitting issue.
I am using logstash-input-beats (3.1.12).
I also see below

github.com/logstash-plugins/logstash-input-beats

Rework the NioEventLoop handling in the beats input

We were using two differents NioEventLoop in the beats input, this was
causing problems in the context of back pressure. When the NioEventLoop
responsable of inserted to the queue was blocked we were still accepting
new connection instead of refusing them. Creating a large amount of
socket in the `CLOSE_WAIT` state.

`ClOSE_WAIT` state could also happen when an exception was raised,
because we were flushing to the queue before closing the connection, now
we close the connection first and flush to the queue. This can result in
a blocked thread for some time but will reduce the connection.
This problem should only happen when using a buffering codec like the
multiline codec.

We also fix an issue when the Main NioEventLoop was blocked, the keep
alive signal were not send back to the client, making the beats handlers
run into a separate executor allow the loop to send the keep alive
signal.

Fixes #272

by ph on 04:50PM - 01 Dec 17

changed 6 files with 120 additions and 65 deletions.

Not sure if I should pull those files and see.

oraclept · February 6, 2018, 10:30pm

One of the ES index was in red state. Fixing ES index resolved close_wait issue on logstash and I/O timeout on filebeat.

system · March 6, 2018, 10:30pm

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.

Topic		Replies	Views
Logstash-input-beats connections doesn't close properly - Am I doing something wrong? Logstash	1	741	October 29, 2017
Logstash disconnects beat after 15 seconds Logstash	2	747	December 26, 2016
Logstash connections are going to close_wait intermittently Logstash	1	992	December 26, 2017
Logstash stops to receive new inputs after 5 minutes Logstash	3	816	October 22, 2018
Problem of using input{filebeat} on logstash Logstash	5	2527	July 6, 2017

Logstash CLOSE_WAIT and Filbe beat I/0 timeout

Related topics