I recently started noticing in my Filebeat logs the following:
DBG connect
ERR Connecting error publishing events (retrying): read tcp <filebeat-host>:43318-><logstash-host>:5044: read: connection reset by peer
DBG send fail
I'm running Filebeat v 5.3.0 and Logstash v 5.2.0 on AWS EC2 instances on Amazon Linux. My instances running Filebeat are behind a load balancer and my Logstash instance is not behind a load balancer, and the instances are in the same VPC. Security groups are open.
filebeat.yml
:
filebeat:
prospectors:
# elasticsearch logs
- input_type: log
paths:
- /var/log/elasticsearch/*.log
registry_file: /var/lib/filebeat/registry
output:
logstash:
hosts: ["elk-logstash:5044"]
bulk_max_size: 4096
timeout: 60
ssl:
insecure: true
logging:
to_files: true
files:
path: /var/log/filebeat
name: elk
rotateeverybytes: 10485760 # = 10MB
keepfiles: 7
level: debug
logstash.yml
is completely vanilla.
/etc/logstash/conf.d/02-beats-input.conf
:
input {
beats {
port => 5044
ssl => false
include_codec_tag => false
client_inactivity_timeout => 600
}
}
I've tried modifying client_inactivity_timeout
, bulk_max_size
, timeout
, and other settings. I am still getting logs to Logstash/Elasticsearch from other instances with similar configs.
I do notice that upon restarting filebeat with service filebeat restart
, filebeat logs seem to show lots of published logs before getting to the connection error:
2017-04-17T15:07:46Z DBG Publish: {
"@timestamp": "2017-04-17T15:07:41.143Z",
"beat": {
"hostname": "[redacted]",
"name": "[redacted]",
"version": "5.3.0"
},
"input_type": "log",
"message": "[redacted]",
"offset": 24733,
"source": "/var/log/elasticsearch/elk-staging_deprecation.log",
"type": "log"
}
2017-04-17T15:07:46Z DBG output worker: publish 237 events
2017-04-17T15:07:46Z DBG connect
2017-04-17T15:07:46Z ERR Connecting error publishing events (retrying): read tcp 10.0.4.25:43624->10.0.2.38:5044: read: connection reset by peer
2017-04-17T15:07:46Z DBG send fail
I'd love any advice that could help me out!