# Filebeat log fails to publish events **URL:** \ **Category:** Beats\ **Tags:** filebeat\ **Created:** [January 27, 2023, 5:12pm UTC](https://discuss.elastic.co/t/filebeat-log-fails-to-publish-events/324127 "2023-01-27T17:12:30Z")\ **Posts on this page:** 11\ **Page:** 1
**Author:** ![zaeemmasood](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/zaeemmasood/32/102383_2.png) [@zaeemmasood](https://discuss.elastic.co/u/zaeemmasood)\ **Post date:** [January 27, 2023, 5:12pm UTC](https://discuss.elastic.co/t/filebeat-log-fails-to-publish-events/324127/1 "2023-01-27T17:12:30Z")
Hello, Our ELK cluster has been stable for a long time, but recently we have started seeing the following error in the filebeat log: ```auto ERROR logstash/async.go:256 Failed to publish events caused by: write tcp xx.xx.xx.xx:12344->xxx.xx.xx.xx:5044: write: connection reset by peer ERROR pipeline/output.go:121 Failed to publish events: write tcp xxx.xx.xx.xx:44834->xxx.xxx.xx.xx.:5044: write: connection reset by peer ``` This results in a lag in messages getting displayed in Kibana. There is no firewall between filebeat and logstash servers. Please guide ---
**Author:** ![zaeemmasood](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/zaeemmasood/32/102383_2.png) [@zaeemmasood](https://discuss.elastic.co/u/zaeemmasood)\ **Post date:** [January 27, 2023, 8:53pm UTC](https://discuss.elastic.co/t/filebeat-log-fails-to-publish-events/324127/2 "2023-01-27T20:53:31Z")
This also followed by the i/o error at times: ```auto 2023-01-27T15:40:24.055-0500 ERROR pipeline/output.go:121 Failed to publish events: write tcp xxx.xx.xx.xxx:32886->xxx.xx.xx.xx:5044: write: connection reset by peer 2023-01-27T15:40:24.103-0500 ERROR pipeline/output.go:121 Failed to publish events: write tcp xxx.xx.xx.xxx:36020->xxx.xx.xx.xxx:5044: write: connection reset by peer 2023-01-27T15:45:49.249-0500 ERROR logstash/async.go:256 Failed to publish events caused by: read tcp xxx.xx.xx.xxx:46912->xxx.xx.xx.xxx:5044: i/o timeout 2023-01-27T15:45:59.114-0500 ERROR logstash/async.go:256 Failed to publish events caused by: client is not connected 2023-01-27T15:46:00.735-0500 ERROR pipeline/output.go:121 Failed to publish events: client is not connected ``` ---
**Author:** ![leandrojmp](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/leandrojmp/32/107231_2.png) [@leandrojmp](https://discuss.elastic.co/u/leandrojmp)\ **Post date:** [January 28, 2023, 2:09pm UTC](https://discuss.elastic.co/t/filebeat-log-fails-to-publish-events/324127/3 "2023-01-28T14:09:36Z")
Did you check your Logstash logs? Your logs suggest that there is some issue in your Logstash machine as the connections are being closed, you need to look at the logs of your logstash and also the system itself for hints of what may be the issue. ---
**Author:** ![zaeemmasood](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/zaeemmasood/32/102383_2.png) [@zaeemmasood](https://discuss.elastic.co/u/zaeemmasood)\ **Post date:** [January 28, 2023, 2:44pm UTC](https://discuss.elastic.co/t/filebeat-log-fails-to-publish-events/324127/4 "2023-01-28T14:44:49Z")
Thanks for responding @leandrojmp I enabled debug on logstash side but due to a very large number of messages pouring in from all sides it kept rolling very fast. Without enabling debug the log is silent. ---
**Author:** ![Rios](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/rios/32/95745_2.png) [@Rios](https://discuss.elastic.co/u/Rios)\ **Post date:** [January 28, 2023, 4:59pm UTC](https://discuss.elastic.co/t/filebeat-log-fails-to-publish-events/324127/5 "2023-01-28T16:59:29Z")
It's like you are using http on https connection, and LS reject connection. What are your settings for `output.logstash` in filebeat.yml? [Here](https://www.elastic.co/guide/en/beats/filebeat/current/publishing-ls-fails-connection-reset-by-peer.html) is another sugestion. ---
**Author:** ![zaeemmasood](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/zaeemmasood/32/102383_2.png) [@zaeemmasood](https://discuss.elastic.co/u/zaeemmasood)\ **Post date:** [January 28, 2023, 5:28pm UTC](https://discuss.elastic.co/t/filebeat-log-fails-to-publish-events/324127/6 "2023-01-28T17:28:03Z")
Thanks for the suggestion. There is no firewall between filebeat and logstash so this might not be relevant. This set up has been running for over 2 years now without issues and this happened all of a sudden. Also logstash is extensively used as it is fed from different sources (client\_inactivity\_timeout also not relavent perhaps). filebeat output is load balanced between 8 logstash hosts as follows: ```auto output.logstash: hosts: ["xxx:xx.xx.xx:5044","xxx:xx.xx.xx:5044","xxx:xx.xx.xx:5044","xxx:xx.xx.xx:5044","xxx:xx.xx.xx:5044","xxx:xx.xx.xx:5044","xxx:xx.xx.xx:5044","xxx:xx.xx.xx:5044"] loadbalance: true index: filebeat logging.level: Info logging.to_files: true logging.files: path: /opt/tal/rtal/elkagent/log/ name: filebeat_tp.log keepfiles: 7 permissions: 0644 ``` ---
**Author:** ![Rios](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/rios/32/95745_2.png) [@Rios](https://discuss.elastic.co/u/Rios)\ **Post date:** [January 28, 2023, 5:34pm UTC](https://discuss.elastic.co/t/filebeat-log-fails-to-publish-events/324127/7 "2023-01-28T17:34:06Z")
Check [this](https://discuss.elastic.co/t/filebeat-error-failed-to-publish-events-caused-connection-reset-by-peer/268400), similar problem ---
**Author:** ![zaeemmasood](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/zaeemmasood/32/102383_2.png) [@zaeemmasood](https://discuss.elastic.co/u/zaeemmasood)\ **Post date:** [January 28, 2023, 5:52pm UTC](https://discuss.elastic.co/t/filebeat-log-fails-to-publish-events/324127/8 "2023-01-28T17:52:00Z")
thanks. Looks like this parameter `ttl` is useful when the logstash hosts are behind a Load balancer, which is not the case. Here hosts defined are individual nodes on port 5044 I can still try to introduce the `ttl` parameter (maybe with a 2 secs time). ---
**Author:** ![leandrojmp](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/leandrojmp/32/107231_2.png) [@leandrojmp](https://discuss.elastic.co/u/leandrojmp)\ **Post date:** [January 29, 2023, 1:33pm UTC](https://discuss.elastic.co/t/filebeat-log-fails-to-publish-events/324127/9 "2023-01-29T13:33:33Z")
> [@zaeemmasood](#): > > filebeat output is load balanced between 8 logstash hosts as follows Did you check the logs on all the 8 nodes? Does your connection error happens randomly or there is one or more logstash hosts that give you more errors? > [@zaeemmasood](#): > > This set up has been running for over 2 years now without issues and this happened all of a sudden. Did you addedd any new kind of data, changed any logstash pipeline recently? Sometimes small changes on a logstash pipeline can lead to performance issues. Your main issue is a network issue and sometimes it is not easy to troubleshoot. Do you have any monitoring on your logstash instances and hosts? You may try to see how is the CPU/memory usage of your instances/hosts when this issue is logged. ---
**Author:** ![zaeemmasood](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/zaeemmasood/32/102383_2.png) [@zaeemmasood](https://discuss.elastic.co/u/zaeemmasood)\ **Post date:** [February 1, 2023, 2:23pm UTC](https://discuss.elastic.co/t/filebeat-log-fails-to-publish-events/324127/10 "2023-02-01T14:23:15Z")
Thanks @leandrojmp. I also suspect a network issue here. For some reason now kibana seems to be updating the messages with current timestamp though the `io timeout` and `write: connection reset by peer` still seem to appear. It still remains a mystery 🙂 and I will keep montoring...... ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [March 1, 2023, 4:23pm UTC](https://discuss.elastic.co/t/filebeat-log-fails-to-publish-events/324127/11 "2023-03-01T16:23:52Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.