Filebeat stopped frequently when sending to Logstash

Elastic Stack Beats
1

Hi Team,

I have installed file beat 5.5.1 version in production server..

Filebeat was worked properly till yesterday

But today filebeat is stopping suddenly without any error..

Once I restart the file beat..it's shipping logs for 10 min's and then it got stopped..

Kindly help...

Regards,
Mugil

2

Please find the below the error log..

2018/06/28 03:05:22.847267 output.go:109: DBG output worker: publish 1019 events

2018/06/28 03:06:53.057825 sync.go:85: ERR Failed to publish events caused by: write tcp 10.10.xx.xx:51050->10.10.xx.xx:5044: write: connection reset by peer

2018/06/28 03:06:53.059190 single.go:91: INFO Error publishing events (retrying): write tcp 10.10.xx.xx:51050->10.10.xx.xx:5044: write: connection reset by peer

1 Like
3

Is your Logstash server up-to-date with the latest version of the beats input? See https://www.elastic.co/guide/en/beats/libbeat/5.6/logstash-installation.html#logstash-input-update. There have been some bug fixes to the input.

These errors could mean there is back-pressure coming from one of your outputs in Logstash. Have you checked the logs for Logstash? Do you have any monitoring in place for Logstash or the destination service to see what kind of throughput you are getting and whether things are operating normally.

4

Thank you so much Andrew..
I have fixed the issue..Now I am not getting any error..

But the filebeat is getting stopped after 5 minutes without any error.
Once I restart the file beat..it's shipping logs for 5 min's and then it got stopped..

Kindly help...

Regards,
Mugil

5

So by upgrading the input plugin you fixed the keep-alive issue that causes the connection between beats and Logstash to be closed. But you still need to address the underlying issue in Logstash which is most likely some form of back-pressure (either a blocked/slow output, a poor performing logstash filter).

If you haven't checked the Logstash logs do that first.

Then take a look at the node stats for the instance. Perhaps you will spot a section in the pipeline that's taking lot's of time or has errors.

Another tool is the pipeline viewier which gives you a visualization of the pipeline with metrics about each stage.

6

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.