Filebeat -> Logstash connection reset by peer, send fail

[reedflinch (Steve Flinchbaugh)]

I recently started noticing in my Filebeat logs the following:

DBG  connect
ERR Connecting error publishing events (retrying): read tcp <filebeat-host>:43318-><logstash-host>:5044: read: connection reset by peer
DBG  send fail

I'm running Filebeat v 5.3.0 and Logstash v 5.2.0 on AWS EC2 instances on Amazon Linux. My instances running Filebeat are behind a load balancer and my Logstash instance is not behind a load balancer, and the instances are in the same VPC. Security groups are open.

filebeat.yml:

filebeat:
  prospectors:
    # elasticsearch logs
    - input_type: log
      paths:
        - /var/log/elasticsearch/*.log

  registry_file: /var/lib/filebeat/registry

output:
  logstash:
    hosts: ["elk-logstash:5044"]
    bulk_max_size: 4096
    timeout: 60
    ssl:
      insecure: true

logging:
  to_files: true
  files:
    path: /var/log/filebeat
    name: elk
    rotateeverybytes: 10485760 # = 10MB
    keepfiles: 7
  level: debug

logstash.yml is completely vanilla.

/etc/logstash/conf.d/02-beats-input.conf:

input {
  beats {
    port => 5044
    ssl => false
    include_codec_tag => false
    client_inactivity_timeout => 600
  }
}

I've tried modifying client_inactivity_timeout, bulk_max_size, timeout, and other settings. I am still getting logs to Logstash/Elasticsearch from other instances with similar configs.

I do notice that upon restarting filebeat with service filebeat restart, filebeat logs seem to show lots of published logs before getting to the connection error:

2017-04-17T15:07:46Z DBG  Publish: {
  "@timestamp": "2017-04-17T15:07:41.143Z",
  "beat": {
    "hostname": "[redacted]",
    "name": "[redacted]",
    "version": "5.3.0"
  },
  "input_type": "log",
  "message": "[redacted]",
  "offset": 24733,
  "source": "/var/log/elasticsearch/elk-staging_deprecation.log",
  "type": "log"
}
2017-04-17T15:07:46Z DBG  output worker: publish 237 events
2017-04-17T15:07:46Z DBG  connect
2017-04-17T15:07:46Z ERR Connecting error publishing events (retrying): read tcp 10.0.4.25:43624->10.0.2.38:5044: read: connection reset by peer
2017-04-17T15:07:46Z DBG  send fail

I'd love any advice that could help me out!

[reedflinch (Steve Flinchbaugh)]

UPDATE: I've found one issue: on my instances that are functioning properly, I am running Filebeat v1.3.1

After downgrading from 5.3.0 to 1.3.1, logs are being shipped properly to Logstash. I still don't know what is breaking on 5.3.0.

[andrewkroh]

You have enabled SSL in Filebeat, but disabled it in Logstash. So the two cannot talk.