Yet another connection reset by peer

[hueyg]

First I have red through pages of these posted and there are actually few applicable proposed solutions. I have tried them and still experience the issue.

Keep in mind that this is a new ELK turnup on RH 7.5 with Nginx. Proxying port 80. All systems (client/host) are running this same level of RH and should be patched to the same version. I have three systems sending filebeats to the ELK server. Two work with no problem whatsoever but one is exhibiting this connection reset by peer problem.

WORKING (x2)
filebeat version 6.4.1 (amd64), libbeat 6.4.1

NONWORKING
filebeat version 6.4.2 (amd64), libbeat 6.4.2

logstash 6.4.1

There are no firewall issues. I can telnet from the symptomatic client to port 5443 on ELK with no issue

telnet 100.100.100.131 5443
Trying 100.100.100.131...
Connected to 100.100.100.131.
Escape character is '^]'.

Filebeat.yml

filebeat.inputs:

- type: log

  # Change to true to enable this input configuration.
  enabled: true

  # Paths that should be crawled and fetched. Glob based paths.
  paths:
    - /var/log/haproxy.log
    - /var/log/oldlogs/haproxy-info.log-20181001
  fields_under_root: true
  fields:
    type: haproxy

output.logstash:

  hosts: ["100.100.100.131:5443"]
  bulk_max_size: 1024
  ssl.certificate_authorities: ["/etc/pki/tls/certs/logstash-forwarder.crt"]
  template.name: "filebeat"
  template.path: "filebeat.template.json"
  template.overwrite: false

Here is my logstash config:

input {
  beats {
    port => 5443
    ssl => true
    ssl_certificate => "/etc/pki/tls/certs/logstash-forwarder.crt"
    ssl_key => "/etc/pki/tls/private/logstash-forwarder.key"
    client_inactivity_timeout => "900"
  }
}

  if [type] == "haproxy" {
    grok {
      match => {"message" => "%{HAPROXYTCP}"}
#      remove_field =>  "message"
    }
  }

output {
  elasticsearch { hosts => ["localhost:9200"]
    hosts => "localhost:9200"
    manage_template => false
    index => "%{[type]}-%{+YYYY.MM.dd}"    
  }
}

So I see the one "client" that is having the issue is running a slighter newer version of filebeat, but see no other difference between the systems.

[hueyg]

The version mismatch was the issue. Honestly I should have checked sooner but the installation was just in the span of a few days. I rolled back to filbeat 6.4.1 and everything is working as expected. Sorry for wasting anyone's time on this.