Filebeat 7.10.2: connection reset by peer Error flooding in filebeat log

[epadmav]

Hi,

We have a cluster in which we're inconsistently observing the below errors filbeat.log within time periods of 5mins/10mins/15mins so on, when using filebeat to send logs to logstash.

2023-11-23T04:40:07.524+0100 ERROR [logstash] logstash/async.go:280 Failed to publish events caused by: write tcp 214.5.244.8:44990->214.5.244.7:5046: write: connection reset by peer
2023-11-23T04:40:08.989+0100 ERROR [publisher_pipeline_output] pipeline/output.go:180 failed to publish events: write tcp 214.5.244.8:44990->214.5.244.7:5046: write: connection reset by peer
2023-11-23T05:20:06.273+0100 ERROR [logstash] logstash/async.go:280 Failed to publish events caused by: write tcp 214.5.244.8:51342->214.5.244.6:5046: write: connection reset by peer
2023-11-23T05:20:07.667+0100 ERROR [publisher_pipeline_output] pipeline/output.go:180 failed to publish events: write tcp 214.5.244.8:51342->214.5.244.6:5046: write: connection reset by peer
2023-11-23T05:50:06.940+0100 ERROR [logstash] logstash/async.go:280 Failed to publish events caused by: write tcp 214.5.244.8:49108->214.5.244.6:5046: write: connection reset by peer
2023-11-23T05:50:08.620+0100 ERROR [publisher_pipeline_output] pipeline/output.go:180 failed to publish events: write tcp 214.5.244.8:49108->214.5.244.6:5046: write: connection reset by peer
2023-11-23T06:20:07.473+0100 ERROR [logstash] logstash/async.go:280 Failed to publish events caused by: write tcp 214.5.244.8:55956->214.5.244.6:5046: write: connection reset by peer
2023-11-23T06:20:09.024+0100 ERROR [publisher_pipeline_output] pipeline/output.go:180 failed to publish events: write tcp 214.5.244.8:55956->214.5.244.6:5046: write: connection reset by peer
~
We already have "client_inactivity_timeout => 900" set in logstash configuration and have below ttl option set in filebeat.yml

Filebeat.yml

output:
  logstash:
    hosts: ["<host1>,<host2>,<host3>"]
    loadbalance: true
    ttl: 60

These are the filebeat and logstash version we're using:
filebeat version - filebeat-7.10.2-1.x86_64
logstash version - logstash-oss-8.4.0-1.x86_64

Could you please help here.

Thanks in advance

[Ljapunov]

Can you show your Logstash Input definition as well please?

Is logstash receiving logs despite the errorlogs? (I am used to have these Errorlines in filebeat-logs from time to time)

[epadmav]

Hi,
Thanks for looking into the issue.
Please find the logstash input configuration:

input {
  beats {
    port => 5046
    client_inactivity_timeout => 900
    ssl => true
    ssl_certificate_authorities => "xxxx"
    ssl_certificate => "xxxx"
    ssl_key => "xxxx"
    ssl_verify_mode => force_peer
    tls_min_version => 1.2
  }
  tcp {
    id => "rsyslog-port"
    port => 6514
    type => "rsyslog-ssl"
    add_field => [ "xxxx", "xxxx" ]
    ssl_enable => true
    ssl_certificate_authorities => "xxxx"
    ssl_cert => "xxxx"
    ssl_key => ""
    ssl_verify => true
  }
}

Yes, Logstash is receiving logs despite the errorlogs in filebeat.