[Solved] Filebeat -> Logstash : connection reset by peer

[CyrilD (Cyril DUBUS)]

Hi there !

I'm testing a pipe like that for dumping my logs : filebeat -> logstash -> elasticsearch and I have strange errors from filebeat :

2017-05-24T18:08:51+02:00 DBG  Try to publish 2 events to logstash with window size 105
2017-05-24T18:08:51+02:00 DBG  handle error: read tcp 172.17.1.5:45543->172.17.105.2:5044: read: connection reset by peer
2017-05-24T18:08:51+02:00 DBG  0 events out of 2 events sent to logstash. Continue sending
2017-05-24T18:08:51+02:00 DBG  close connection
2017-05-24T18:08:51+02:00 DBG  closing
2017-05-24T18:08:51+02:00 ERR Failed to publish events caused by: read tcp 172.17.1.5:45543->172.17.105.2:5044: read: connection reset by peer
2017-05-24T18:08:51+02:00 INFO Error publishing events (retrying): read tcp 172.17.1.5:45543->172.17.105.2:5044: read: connection reset by peer
2017-05-24T18:08:51+02:00 DBG  close connection
2017-05-24T18:08:51+02:00 DBG  send fail

These errors are happening every 3~5 minutes. I'm only dumping /var/log/syslog on a quiet host (2 or 3 lines per minutes) so I don't think there is an overload.

As you can see, my filebeat host has for IP 172.17.1.5 and my logstash host has for IP 172.17.105.2
I did a lot of network tests between my two hosts and I'm pretty sure there is no problems at all on network side.

Filebeat configuration is like that :

filebeat:
  prospectors:
  - document_type: syslog
    exclude_lines: [snmpd+(.)*(Connection from UDP)+]
    input_type: log
    paths: [/var/log/syslog]

output:
  logstash:
    hosts: ["logbucket01:5044"]

logging:
  to_syslog: false
  level: debug

And the relevant Logstash configuration is like that :

input { 
    beats {
        port => 5044
    }
}

Here are my versions :

filebeat_host# /usr/share/filebeat/bin/filebeat --version 
filebeat version 5.4.0 (amd64), libbeat 5.4.0

logstash_host# /usr/share/logstash/bin/logstash --version
logstash 5.4.0

I installed both of them with the official deb files.

I did not see any useful log on logstash side, even in debug mode there is no problem at all. I don't see anything in any system logs on both hosts.

Does anybody has an idea on how to solve that ? I can't imagine to go further with such errors on a simple case like that. I must have done something wrong somewhere but I have no idea where to look. I read a few post on the same errors but they are either abandoned or concerning something I dont use (ssl, tls..). So .. no clues.

If you need more informations just tell me I'll provide everything useful to solve this.

Thanks a lot,

Cyril

[andrewkroh]

Logstash could be resetting the connection due to inactivity, in which case this shouldn't be a problem. You can try increasing the client_inactivity_timeout on the LS side.

But from the logs, it looks like it's happening when sending. This could mean there is something blocking the LS pipeline. Does the problem occur if you output only to stdout or to a file from LS?

[CyrilD (Cyril DUBUS)]

Hi Andrew,

Thanks for your answer, I'll try to increase the client_inactivity_timeout to see if the error still occur. This settings looks promising !

I don't understand what you mean by "output only to stdout or to a file from LS?". Do you mean I should drop the elasticsearch output and replace it with a stdout output to test if the problem still occur ? Because since I get the error on the filebeat side, and nothing at all in logstash logs I doubt the elasticsearch output is the problem so I'm not sure I get your idea (my english is not very good). If the inactivity timeout has no effect, I will give it a try even if I'm not sure I understood correctly because it's easy to do.

I'll let you know the results, let me know if I get the second part of your message.

Thanks again !

[CyrilD (Cyril DUBUS)]

Hi Andrew,

I think the settings did the trick. It's very weird because I changed the settings, restarted logstash and the problem occurred again during 5 or 6 minutes, then it stopped and now that's a few hours I have no errors anymore.

I also upgraded my kernel of my logstash server but I don't think that was the root cause.

I did not try the output to stdout since the problem seems to be resolved.

Thanks for your time,

Cyril

[andrewkroh]

Using stdout instead of ES would have helped if there was a problem with ES output. For example, if ES was being overloaded or slow, the backpressure could eventually cause Logstash to sever the connections to Beats.