Filebeat TLS connect to Logstash fails

Elastic Stack Beats
1

Hi all!

We've set up Logstash to collect Filebeat data via the beats plugin:

input {
  beats {
    port => 5044
    ssl => true
    ssl_certificate_authorities => ["/etc/logstash/certs/ca.cert.pem"]
    ssl_certificate => "/etc/logstash/certs/server.cert.pem"
    ssl_key => "/etc/logstash/server.pem"
    ssl_verify_mode => "force_peer"
    type => "logs"
  }
}

Logstash starts using this configuration (filter section is empty for now, all output goes to stdout). The Logstash server is on a different network segment than the Filebeat server so we opened the firewall to let Filebeat traffic pass. We've verified with telnet that we can actually connect to the Logstash server. A tcpdump on the Logstash server shows that packets arrive and everything's fine. Filebeat is configured like this in the output section:

output:
  logstash:
    hosts: ["server.fqdn.here:5044"]
    tls:
      certificate_authorities: ["/etc/filebeat/ca.cert.pem"]
      certificate: "/etc/filebeat/filebeat-client.cert.pem"
      certificate_key: "/etc/filebeat/filebeat-client.key.pem"

When we start Filebeat with this configuration, we get the following error messages:

transport.go:125: ERR SSL client failed to connect with: EOF
single.go:126: INFO Connecting error publishing events (retrying): EOF
single.go:152: INFO send fail
single.go:159: INFO backoff retry: 16s

We can see in the tcpdump output that packets are exchanged between Logstash and Filebeat servers, however, we don't see why the connection fails. All the solutions in other posts in the forum to the "ERR SSL client failed to connect with: EOF" error don't seem to work for us. Also, changing

ssl_verify_mode => "force_peer"

to

ssl_verify_mode => "peer"

didn't work either. Please note that we have FQDNs in the certificates (own private CA without Intermediate CAs) and use the FQDN in the Filebeat configuration. We don't have SANs in the certificates.

We have the following software versions installed:

  • Logstash: 2.2.4,
  • Beats input plugin: 2.2.8,
  • Filebeat: 1.2.1

It'd be great if someone could point us in the right direction. Running Logstash with "-v --debug --verbose" didn't yield any useful output. Any help would be highly appreciated. Thanks!

cs1

2

out of curiosity, when doing tcpdump which side did send the FIN flag.

Let's try step by step. First tcp only. Next ssl enabled, but client auth disabled.

3

Hi Steffen,

thanks for your reply. Unfortunately I didn't capture any pcap files so I don't have the output anymore right know (please read on why).

I think that we "solved" the problem. We used a private CA because we thought that we couldn't use our PKI since we read that signed certs from intermediate CAs can't be used with Filebeat. After having trouble with the certificates from our private CA (we still don't know what the problem is because the SSL connection using "openssl s_client -connect ..." to connect to Logstash worked) we simply tried certificates from our PKI that we already had on the machines and that were signed by an intermediate CA. And to our surprise, that works like a charm. As CA file we use a file containing the whole chain of the Root CA - Intermediate CA - RA and we also have that chain embedded in the signed certs. So now everything's even better than before because now we can use globally valid CA certs. Still, I have no idea whatsoever what Filebeat's problem was with the original certificates because our openssl connection test worked.

4

It's kinda funny. I head jruby (logstash side) having problems with intermediate CAs, but not sure.

5

We use intermediate CA signed certs on both sides. Everything seems to be working nicely so far. And ssl_verify_mode => "force_peer" is set so my guess is that everything's in order TLS-wise.