Filebeat is not connecting logstash after the upgrade

[harish.g (harish)]

We are using filbeat version 5.* which is not able to connect to logstash version 5.* after the upgrade .
During the previous version of file beat 1.* and logstash is 2.* able to establish the connection but after the upgrade its not working . Seeing below error file beat logs
017-09-07T09:23:55-05:00 INFO Non-zero metrics in the last 30s: libbeat.logstash.call_count.PublishEvents=1 libbeat.logstash.publish.read_bytes=12 libbeat.logstash.publish.read_errors=1 libbeat.logstash.publish.write_bytes=3482 libbeat.logstash.published_and_acked_events=18 libbeat.logstash.published_but_not_acked_events=1727
2017-09-07T09:24:02-05:00 ERR Connecting error publishing events (retrying): dial tcp 10.190.157.83:5044: getsockopt: connection refused
2017-09-07T09:24:18-05:00 ERR Connecting error publishing events (retrying): dial tcp 10.190.157.83:5044: getsockopt: connection refused
2017-09-07T09:24:25-05:00 INFO No non-zero metrics in the last 30s
2017-09-07T09:24:50-05:00 ERR Connecting error publishing events (retrying): dial tcp 10.190.157.83:5044: getsockopt: connection refused
2017-09-07T09:24:55-05:00 INFO No non-zero metrics in the last 30s
2017-09-07T09:25:25-05:00 INFO No non-zero metrics in the last 30s
2017-09-07T09:25:51-05:00 ERR Failed to publish events caused by: read tcp 10.160.13.245:61227->10.190.157.83:5044: read: connection reset by peer

PFA for the filebeat configuration and no errors from the logs . Please advise on the same

[steffens (Steffen Siering)]

Which exact versions did you upgrade to? How many filebeat instances have you send to one Logstash instance?

The getsockopt: connection refused message indicates Logstash potentially not running.

This message indicates logstash (or the host) did close the connection while filebeat was waiting for an ACK: 2017-09-07T09:25:51-05:00 ERR Failed to publish events caused by: read tcp 10.160.13.245:61227->10.190.157.83:5044: read: connection reset by peer

[harish.g (harish)]

we have upgraded our ELK version 2.2 to version 5.5* , where as file beat is upgraded from 1.3 to version 5.* . In other environments filebeat is able connect logstash but in production its' not happening .
i don't see any errors in the logstash logs . where as logstash is running properly . May i know what corrective action need to take further on this .

[steffens (Steffen Siering)]

Uhm... I guess exact version of filebeat is 5.5.0? And logstash is at version 5.5.0 as well?

The second message connection reset by peer is not about not being able to connect. It is logstash closing the connection for whatever reason. Logstash closes idle connections. Maybe it's a problem in Logstash assuming the connection being idle, besides Logstash actually being slowed down by its processing or outputs? You can try to increase the client_inactivity_timeout in logstash.