Filebeat -> Logstash connection reset by peer, send fail

I recently started noticing in my Filebeat logs the following:

DBG  connect
ERR Connecting error publishing events (retrying): read tcp <filebeat-host>:43318-><logstash-host>:5044: read: connection reset by peer
DBG  send fail

I'm running Filebeat v 5.3.0 and Logstash v 5.2.0 on AWS EC2 instances on Amazon Linux. My instances running Filebeat are behind a load balancer and my Logstash instance is not behind a load balancer, and the instances are in the same VPC. Security groups are open.

filebeat.yml:

filebeat:
  prospectors:
    # elasticsearch logs
    - input_type: log
      paths:
        - /var/log/elasticsearch/*.log

  registry_file: /var/lib/filebeat/registry

output:
  logstash:
    hosts: ["elk-logstash:5044"]
    bulk_max_size: 4096
    timeout: 60
    ssl:
      insecure: true

logging:
  to_files: true
  files:
    path: /var/log/filebeat
    name: elk
    rotateeverybytes: 10485760 # = 10MB
    keepfiles: 7
  level: debug

logstash.yml is completely vanilla.

/etc/logstash/conf.d/02-beats-input.conf:

input {
  beats {
    port => 5044
    ssl => false
    include_codec_tag => false
    client_inactivity_timeout => 600
  }
}

I've tried modifying client_inactivity_timeout, bulk_max_size, timeout, and other settings. I am still getting logs to Logstash/Elasticsearch from other instances with similar configs.

I do notice that upon restarting filebeat with service filebeat restart, filebeat logs seem to show lots of published logs before getting to the connection error:

2017-04-17T15:07:46Z DBG  Publish: {
  "@timestamp": "2017-04-17T15:07:41.143Z",
  "beat": {
    "hostname": "[redacted]",
    "name": "[redacted]",
    "version": "5.3.0"
  },
  "input_type": "log",
  "message": "[redacted]",
  "offset": 24733,
  "source": "/var/log/elasticsearch/elk-staging_deprecation.log",
  "type": "log"
}
2017-04-17T15:07:46Z DBG  output worker: publish 237 events
2017-04-17T15:07:46Z DBG  connect
2017-04-17T15:07:46Z ERR Connecting error publishing events (retrying): read tcp 10.0.4.25:43624->10.0.2.38:5044: read: connection reset by peer
2017-04-17T15:07:46Z DBG  send fail

I'd love any advice that could help me out!

UPDATE: I've found one issue: on my instances that are functioning properly, I am running Filebeat v1.3.1 :expressionless:

After downgrading from 5.3.0 to 1.3.1, logs are being shipped properly to Logstash. I still don't know what is breaking on 5.3.0.

You have enabled SSL in Filebeat, but disabled it in Logstash. So the two cannot talk.

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.