First I have red through pages of these posted and there are actually few applicable proposed solutions. I have tried them and still experience the issue.
Keep in mind that this is a new ELK turnup on RH 7.5 with Nginx. Proxying port 80. All systems (client/host) are running this same level of RH and should be patched to the same version. I have three systems sending filebeats to the ELK server. Two work with no problem whatsoever but one is exhibiting this connection reset by peer problem.
WORKING (x2)
filebeat version 6.4.1 (amd64), libbeat 6.4.1
NONWORKING
filebeat version 6.4.2 (amd64), libbeat 6.4.2
logstash 6.4.1
There are no firewall issues. I can telnet from the symptomatic client to port 5443 on ELK with no issue
telnet 100.100.100.131 5443
Trying 100.100.100.131...
Connected to 100.100.100.131.
Escape character is '^]'.
Filebeat.yml
filebeat.inputs:
- type: log
# Change to true to enable this input configuration.
enabled: true
# Paths that should be crawled and fetched. Glob based paths.
paths:
- /var/log/haproxy.log
- /var/log/oldlogs/haproxy-info.log-20181001
fields_under_root: true
fields:
type: haproxy
output.logstash:
hosts: ["100.100.100.131:5443"]
bulk_max_size: 1024
ssl.certificate_authorities: ["/etc/pki/tls/certs/logstash-forwarder.crt"]
template.name: "filebeat"
template.path: "filebeat.template.json"
template.overwrite: false
Here is my logstash config:
input {
beats {
port => 5443
ssl => true
ssl_certificate => "/etc/pki/tls/certs/logstash-forwarder.crt"
ssl_key => "/etc/pki/tls/private/logstash-forwarder.key"
client_inactivity_timeout => "900"
}
}
if [type] == "haproxy" {
grok {
match => {"message" => "%{HAPROXYTCP}"}
# remove_field => "message"
}
}
output {
elasticsearch { hosts => ["localhost:9200"]
hosts => "localhost:9200"
manage_template => false
index => "%{[type]}-%{+YYYY.MM.dd}"
}
}
So I see the one "client" that is having the issue is running a slighter newer version of filebeat, but see no other difference between the systems.