Logs seem to be flowing OK but constant "connection reset by peer" errors in the filebeat log

Elastic Stack Beats
#1

filebeat 5.2.2

pertinent part of the config:

...
output.logstash:
hosts: ["ps-dev-elk.domain.com:5044"]
ssl.certificate_authorities: ["/etc/pki/tls/certs/filebeat.crt"]
timeout: 30
...

logs seem to be flowing fine from this node to our logstash server however I'm frequently seeing errors like this in the filebeat log:

2017-03-21T21:34:35Z INFO Error publishing events (retrying): write tcp 10.140.50.31:51212->10.140.40.206:5044: write: connection reset by peer

and I'm not seeing any matching errors or issues in the logs on the logstash server. What might be causing this?

#2

There are quite a few similar issues already on discuss. Best check this entries to see if one helps you solve the problem: https://discuss.elastic.co/search?q=Error%20publishing%20events If not, let us know and we can dig deeper.

#3

@ruflin I took a look through quite a few of those and they don't seem to be related to what I'm seeing. All of those seemed to have no success whatsoever sending lots from filebeat to LS. My logs are actually making it through but I'm also seeing these intermittent errors on a regular basis. Perhaps I need to set some tuning parameters or something like that?

#4

Filebeat will keep resending the events until they get confirmed. It seems sometimes Logstash does not respond fast enough. That could be because of some network errors or that Logstash is too busy. Do you see it during peak load times? As long as the error is not constant it means from time to time there are issue but filebeat will make sure all events are delivered.

I would expect you to see some errors or at least info somewhere in the LS logs. Do you have something between FB and LS?

#5

Logstash version? logstash-input-beats plugin version? Logstash closes inactive connections, leading filebeat to reconnect and continue sending logs. Some logstash-input-beats plugin versions seem to have a bug of closing an active connection, though. I think upgrading/downgrading logstash-input-beats plugin to version 3.1.10 should help (given you're using logstash 5.x).

#6

I just took a look and I'm seeing these about once a minute or once every other minute, consistently.

I'm running logstash 5.2.2-1 installed from the elastic repos. Based on the logs it looks like we are running logstash-input-beats 3.1.12.

How do I downgrade to 3.1.10 and is that something I should try?

#7

The plugin closes inactive connections. The timeout for inactive connections can be adjusted in logstash. Maybe increasing the timeout to some much bigger value will help.

checkout bin/logstash-plugin tool for installing/uninstalling/updating plugins. As far as I can tell from other user reports, 3.1.10 seems to be most stable. Checking the changelog you might want to try 3.1.14 first, though. Sorry for the inconvenience, but I haven't run into this problem myself yet and can only recite from other users experiences so far.

#8

I just updated to 3.1.14. I'll let that run for a bit and see how things look.

#9

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.