Hi Folks,
I've been working on upgrading my Ansible playbooks for ELK/EFK to the latest 5.2.x versions and I'm having problems with TLS/SSL connections from filebeat --> logstash. For the purpose of this post I am setting up everything manually until it all works.
Everything works as it should with non-encrypted connections but I get the following error sending from filebeat (5.2.1) to my logstash server (5.2.1). This is all on latest CentOS 7.3.
FILEBEAT SIDE
Here is the error I'm getting in filebeat:
2017-02-16T15:48:50Z DBG End of file reached: /var/log/messages; Backoff now.
2017-02-16T15:48:52Z DBG End of file reached: /var/log/boot.log; Backoff now.
2017-02-16T15:48:52Z DBG End of file reached: /var/log/wpa_supplicant.log; Backoff now.
2017-02-16T15:48:52Z DBG Flushing spooler because of timeout. Events flushed: 0
2017-02-16T15:48:56Z DBG connect
2017-02-16T15:48:56Z ERR Connecting error publishing events (retrying): remote error: tls: handshake failure
2017-02-16T15:48:56Z DBG send fail
Here is my filebeat configuration:
filebeat:
prospectors:
-
paths:
- /var/log/*.log
- /var/log/messages
- /var/log/foreman/*.log
- /var/log/foreman-proxy/*.log
- /var/log/nova/*.log
- /var/log/neutron/*.log
- /var/log/cinder/*.log
- /var/log/keystone/*.log
- /var/log/horizon/*.log
- /var/log/glance/*.log
- /var/log/mariadb/*.log
- /var/log/rabbitmq/*.log
- /var/log/mongodb/*.log
- /var/log/ceilometer/*.log
- /var/log/ceph/*.log
- /var/log/heat/*.log
- /var/log/openvswitch/*.log
- /var/log/pcsd/*.log
- /var/log/puppet/*.log
- /var/log/redis/*.log
- /var/log/glusterfs/*.log
- /var/log/swift/*.log
input_type: log
document_type: syslog
output:
logstash:
hosts: ["192.168.122.81:5044"]
bulk_max_size: 1024
ssl:
certificate_authorities: "/etc/filebeat/filebeat-forwarder.crt"
certificate_key: "/etc/filebeat/filebeat-forwarder.key"
supported_protocols: [TLSv1.0, TLSv1.1, TLSv1.2, SSLv3]
min_version: 1.0
max_version: 1.2
shipper:
logging:
files:
selectors: ["*"]
level: debug
LOGSTASH SIDE
Here is the configuration on the logstash side
input {
tcp {
port => 5000
type => syslog
}
udp {
port => 5000
type => syslog
}
}
filter {
if [type] == "syslog" {
grok {
match => { "message" => "%{SYSLOGTIMESTAMP:syslog_timestamp} %{SYSLOGHOST:syslog_hostname} %{DATA:syslog_program}(?:\[%{POSINT:syslog_pid}\])?: %{GREEDYDATA:syslog_message}" }
add_field => [ "received_at", "%{@timestamp}" ]
add_field => [ "received_from", "%{host}" ]
}
date {
match => [ "syslog_timestamp", "MMM d HH:mm:ss", "MMM dd HH:mm:ss" ]
}
}
}
output {
elasticsearch { hosts => ["localhost:9200"] }
stdout { codec => rubydebug }
}
input {
beats {
port => 5044
ssl => true
ssl_certificate_authorities => ["/etc/pki/tls/certs/ca-bundle.crt"]
ssl_certificate => ["/usr/share/logstash/filebeat-forwarder.crt"]
ssl_key => ["/usr/share/logstash/filebeat-forwarder.key"]
ssl_verify_mode => "none"
}
}
SSL Information:
Above in my logstash.conf I have tried both /etc/pki/tls/certs/ca-bundle.crt (after adding the generated certificate to it) and the original filebeat-forwarder.crt for the ssl_certificate_authorities directive with the same result of remote error: tls: handshake failure
I am creating my SSL certificates with AltSANS support via Ansible, and prior 2.x versions worked without an issue (changing the tls to ssl per the 5.x version changes).
For posterity here's the Ansible code that generates them, and md5sum on both locations says it's the same file.
- name: Create client forwarder SSL certificate
command: openssl req -subj '/CN={{ ansible_fqdn }}/' -config /etc/pki/tls/openssl_extras.cnf \
-x509 -days 3650 -batch -nodes -newkey rsa:2048 -keyout /usr/share/logstash/filebeat-forwarder.key \
-out /usr/share/logstash/filebeat-forwarder.crt
ignore_errors: true
when: filebeat_forwarder_ssl_exists != 0
I am specifying the same filebeat-forwarder.crt certificate for any mention of both certificate or CA on both sides and the permissions on all the certificates are liberal (tried 644 and 755). They are located inside accessible locations like logstash $HOME or filebeat $HOME.
I've tried secure=false on the filebeat side and I am using ssl_verify_mode => "none" on the logstash side as well but it doesn't seem to make a difference.
I've also tried adding the filebeat-forwarder.crt to my /etc/pki/tls/certs/ca-bundle.crt on the logstash side.
Any ideas what I am missing? Thank you.