Filebeat Error: "Write: connection reset by peer"

[DougR]

TL;DR

Filebeat 7.4.2 fails write to logstash multiple times per day with write: connection reset by peer

Longer version

Filebeat version is 7.4.2 for Linux.

I see this issue periodically across our environment. Ordinarily, restarting the filebeat resolves the issue. However, we have one specific filebeat, which frequently fails with the following errors to the log:

Nov 05 08:30:51 host.example.com filebeat[32310]: 2019-11-05T08:30:51.924-0600        WARN        beater/filebeat.go:368        Filebeat is unable to load the Ingest Node pipelines for the configured modules because the Elasticsearch output is not configured/enabled. If you have already loaded the Ingest Node pipelines or are using Logstash pipelines, you can ignore this warning.
Nov 05 11:46:11 host.example.com filebeat[32310]: 2019-11-05T11:46:11.677-0600        ERROR        logstash/async.go:256        Failed to publish events caused by: write tcp 10.0.0.62:43034->10.0.0.63:5044: write: connection reset by peer
Nov 05 11:46:13 host.example.com filebeat[32310]: 2019-11-05T11:46:13.127-0600        ERROR        pipeline/output.go:121        Failed to publish events: write tcp 10.0.0.62:43034->10.0.0.63:5044: write: connection reset by peer
Nov 05 13:03:07 host.example.com filebeat[32310]: 2019-11-05T13:03:07.780-0600        ERROR        logstash/async.go:256        Failed to publish events caused by: write tcp 10.0.0.62:47248->10.0.0.63:5044: write: connection reset by peer
Nov 05 13:03:09 host.example.com filebeat[32310]: 2019-11-05T13:03:09.057-0600        ERROR        pipeline/output.go:121        Failed to publish events: write tcp 10.0.0.62:47248->10.0.0.63:5044: write: connection reset by peer

Restarting the filebeat will temporarily resolve this issue, but it generally reoccurs within a few hours.

My filebeat.yml is as follows:

filebeat:
  inputs:
    - type: log
      paths:
        - /path/to/log/*.log
 
      multiline:
        pattern: '^[0-9]{4}-[0-9]{2}-[0-9]{2} [0-9]{2}:[0-9]{2}:[0-9]{2},[0-9]{3}'
        negate: true
        match: after

      fields:
        application:
          name: myapp
          env: prod
        
      fields_under_root: true
      
      # tail_files: true

    - type: log
      paths:
        - /path/to/other/log/*.log

      fields:
        application:
          name: myapp
          env: prod
        
      fields_under_root: true
      
      # tail_files: true

  #registry_file_permissions: 0644

# Configure logging.
# Available log levels are: error, warning, info, debug
logging:
  level: debug
  to_files: true
  files:
    path: /var/log/filebeat
    name: filebeat.log
    keepfiles: 5
    permissions: 0644

  # At debug level, you can selectively enable logging only for some
  # components. To enable all selectors use ["*"]. Examples of other
  # selectors are "beat", "publish", "service".
  # selectors: ["*"]

output:
  logstash:
    index: filebeat
    hosts:
      - logstash.example.com:5044

I've already set the following in the beats.conf file without any results:

input {
    beats {
        # host => "0.0.0.0"
        client_inactivity_timeout => 3000
        port => 5044
    }
}

I've also done a remove/reinstall on the beat without any results.