Let's Encrypt and Filebeat TLS

Elastic Stack Beats
1

Hey there,

I've been following this guide to get an ELK stack up and running - https://www.digitalocean.com/community/tutorials/how-to-install-elasticsearch-logstash-and-kibana-elk-stack-on-ubuntu-14-04

However, I don't want to set up PKI stuff as defined in the guide (skip to Generate SSL Certificates) - I want to use a public certificate, in this case from Let's Encrypt.

They want me to use this command: sudo openssl req -config /etc/ssl/openssl.cnf -x509 -days 3650 -batch -nodes -newkey rsa:2048 -keyout private/logstash-forwarder.key -out certs/logstash-forwarder.crt

So instead I used acme.sh and DNS validation to generate a certificate for my Logstash target - let's call it logstash.company.com - easy, right? Now I have a bunch of certificates:

letsencrypt@elk:~/.acme.sh/logstash.company.com$ ls
ca.cer logstash.company.com.csr
fullchain.cer logstash.company.com.key
logstash.company.com.cer logstash.company.com.ssl.conf
logstash.company.com.conf

In the Logstash config "2-beats-input.conf" (as in the guide) I set ssl_certificate and ssl_key to the keys and cert I just generated:
ssl_certificate => "/etc/logstash/ssl/logstash.company.com.cer"
ssl_key => "/etc/logstash/ssl/logstash.company.com.key"

Logstash appears to act like everything is fine.

Finally, it came time to set up Filebeat and this is where it fell apart.

I copied the cert (logstash.company.com.cer) to another server, which we'll call web1. I configured it as in the guide, and copied the cert to "/etc/pki/tls/certs/logstash-forwarder.crt".

I followed the rest of the guide as-is. I thought it was a little odd, however, that it said to set the certificate as "certificate_authorities: ["/etc/pki/tls/certs/logstash-forwarder.crt"]"

Anyways, I think I've tracked down the issue - regardless of whether or not I set certificate_authorities to the cert, I get this error in Filebeat's logs (I set log level to WARNING):

2016-07-19T13:29:34Z ERR SSL client failed to connect with: x509: certificate signed by unknown authority

It's a standard certificate from Let's Encrypt, which ought to be trusted. Maybe I need to copy ca.cer and use that instead?

Sorry for the sort of rambling text. I'm just tearing my hair out trying to get this to work.

EDIT: I have tried using fullchain.cer instead, but the issue remains - x509: certificate signed by unknown authority. I also double-checked that the cert is using the real CA and not the staging/"fake" CA:
Issuer: C=US, O=Let's Encrypt, CN=Let's Encrypt Authority X3

1 Like
2

So if your Logstash server is using a certificate that is trusted by the OS then you shouldn't even need to use the certificate_authorities option in Filebeat. In theory Filebeat can just use the OS trust store to validate the Let's Encrypt cert.

Does curl -v https://logstash.company.com:5044 say that the certificate is invalid? Or does it say "Empty reply from server" which would mean certificate validation passed. Secure communication with Logstash | Filebeat Reference [8.11] | Elastic

3

Oh, I just remembered that intermediate CAs are a problem for Logstash at the moment. https://github.com/logstash-plugins/logstash-input-beats/issues/64

So you probably need to follow this workaround in Filebeat until it gets fixed: https://github.com/elastic/beats/issues/1494

1 Like
4

Hi @andrewkroh,

Thanks for taking the time :slight_smile:

Yes, I also tried running without certificate authorities set as in theory it shouldn't be necessary. Same issues occurred.

Meanwhile, I tried your curl command on "web1" and got the following output. It's a Debian 8 box.

root@web1:/var/log/mybeat# curl -v https://logstash.company.com:5044

  • Rebuilt URL to: https://logstash.company.com:5044/
  • Hostname was NOT found in DNS cache
  • Trying 10.X.Y.Z...
  • Connected to logstash.admin.ecommistry.com (10.X.Y.Z) port 5044 (#0)
  • successfully set certificate verify locations:
  • CAfile: none
    CApath: /etc/ssl/certs
  • SSLv3, TLS handshake, Client hello (1):
  • SSLv3, TLS handshake, Server hello (2):
  • SSLv3, TLS handshake, CERT (11):
  • SSLv3, TLS alert, Server hello (2):
  • SSL certificate problem: unable to get local issuer certificate
  • Closing connection 0
  • SSLv3, TLS alert, Client hello (1):
    curl: (60) SSL certificate problem: unable to get local issuer certificate
    More details here: curl - SSL CA Certificates

Interestingly, I get a totally different result from my OS X laptop:

rain:~ wings$ curl -v https://logstash.company.com:5044

GET / HTTP/1.1
Host: logstash.company.com:5044
User-Agent: curl/7.43.0
Accept: /

  • Empty reply from server
  • Connection #0 to host logstash.company.com left intact
    curl: (52) Empty reply from server

I will look into the intermediate CA issues and the workarounds. Thanks!

5

Okay, I'm not entirely sure why this works as a fix, but here goes - I simply put ca.cer from the site's Let's Encrypt folder into the configuration:

certificate_authorities: ["/etc/filebeat/ssl/ca.cer"]

And suddenly Filebeat works with full TLS.

To be clear, ca.cer is (I think?) just the root certificate - and fullchain.cer is the main certificate with the root certificate attached to it.

Either that or they are just the intermediate certs and the root cert doesn't come into it at all.

Anyways, that seems to be the fix. I might write a blog post in a few days about using LE and Filebeat as they seem to be a good fit. Thank you so much for your help, I've been struggling with this for hours and hadn't thought to try the ca.cer approach.

6

Hi zorlin

Please write a blog (or at least your steps here) I'm doing exactly the same thing and would like to learn from your mistakes!

Did you create the LE cert for the reverse proxy virtual host?

7

slightly off topic, I assume that if I had standard self signed certs in place (or even SSL disabled as all components are currently on one machine for testing) it's a simple case to just replace the certs with those from LE?

8

Hi @SockThief,

Will definitely do a writeup at some point soon. No, I created it with two hostnames - "logs.company.com" in case I wanted to add SSL to the Kibana (via reverse proxy virtual host) server, and "logstash.company.com" for encrypted Filebeat (and other) traffic.

The plan is to have two servers running the full ELK stack, with a shared ElasticSearch cluster. Then loadbalance logstash.company.com via plain DNS round robin, though I may also use the native loadbalancing in Filebeat instead.

Then to have an Elastic Load Balancer on AWS in front of the Kibana panel, load balancing between the two Kibana reverse proxies.

So, to answer your question - no, I didn't create it for the reverse proxy virtual host, but I did add that as a SAN in case we ever want to change to that setup in the future. For now SSL termination will be done at the ELBs.

Now your second question - if I am understanding you correctly, yes - it should be a fairly simple matter to replace the certificates you generate.

I can't redo this on the production system, so I'll try to replicate it on my personal server and document the process.

9

This topic was automatically closed after 21 days. New replies are no longer allowed.