Filebeat 7.10.2: connection reset by peer Error flooding in filebeat log

Hi,

We have a cluster in which we're inconsistently observing the below errors filbeat.log within time periods of 5mins/10mins/15mins so on, when using filebeat to send logs to logstash.

2023-11-23T04:40:07.524+0100 ERROR [logstash] logstash/async.go:280 Failed to publish events caused by: write tcp 214.5.244.8:44990->214.5.244.7:5046: write: connection reset by peer
2023-11-23T04:40:08.989+0100 ERROR [publisher_pipeline_output] pipeline/output.go:180 failed to publish events: write tcp 214.5.244.8:44990->214.5.244.7:5046: write: connection reset by peer
2023-11-23T05:20:06.273+0100 ERROR [logstash] logstash/async.go:280 Failed to publish events caused by: write tcp 214.5.244.8:51342->214.5.244.6:5046: write: connection reset by peer
2023-11-23T05:20:07.667+0100 ERROR [publisher_pipeline_output] pipeline/output.go:180 failed to publish events: write tcp 214.5.244.8:51342->214.5.244.6:5046: write: connection reset by peer
2023-11-23T05:50:06.940+0100 ERROR [logstash] logstash/async.go:280 Failed to publish events caused by: write tcp 214.5.244.8:49108->214.5.244.6:5046: write: connection reset by peer
2023-11-23T05:50:08.620+0100 ERROR [publisher_pipeline_output] pipeline/output.go:180 failed to publish events: write tcp 214.5.244.8:49108->214.5.244.6:5046: write: connection reset by peer
2023-11-23T06:20:07.473+0100 ERROR [logstash] logstash/async.go:280 Failed to publish events caused by: write tcp 214.5.244.8:55956->214.5.244.6:5046: write: connection reset by peer
2023-11-23T06:20:09.024+0100 ERROR [publisher_pipeline_output] pipeline/output.go:180 failed to publish events: write tcp 214.5.244.8:55956->214.5.244.6:5046: write: connection reset by peer
~
We already have "client_inactivity_timeout => 900" set in logstash configuration and have below ttl option set in filebeat.yml

Filebeat.yml

output:
  logstash:
    hosts: ["<host1>,<host2>,<host3>"]
    loadbalance: true
    ttl: 60

These are the filebeat and logstash version we're using:
filebeat version - filebeat-7.10.2-1.x86_64
logstash version - logstash-oss-8.4.0-1.x86_64

Could you please help here.

Thanks in advance

Can you show your Logstash Input definition as well please?

Is logstash receiving logs despite the errorlogs? (I am used to have these Errorlines in filebeat-logs from time to time)

Hi,
Thanks for looking into the issue.
Please find the logstash input configuration:

input {
  beats {
    port => 5046
    client_inactivity_timeout => 900
    ssl => true
    ssl_certificate_authorities => "xxxx"
    ssl_certificate => "xxxx"
    ssl_key => "xxxx"
    ssl_verify_mode => force_peer
    tls_min_version => 1.2
  }
  tcp {
    id => "rsyslog-port"
    port => 6514
    type => "rsyslog-ssl"
    add_field => [ "xxxx", "xxxx" ]
    ssl_enable => true
    ssl_certificate_authorities => "xxxx"
    ssl_cert => "xxxx"
    ssl_key => ""
    ssl_verify => true
  }
}

Yes, Logstash is receiving logs despite the errorlogs in filebeat.

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.