# Let's Encrypt and Filebeat TLS **URL:** \ **Category:** Beats\ **Tags:** filebeat\ **Created:** [July 19, 2016, 1:32pm UTC](https://discuss.elastic.co/t/lets-encrypt-and-filebeat-tls/55882 "2016-07-19T13:32:39Z")\ **Posts on this page:** 9\ **Page:** 1
**Author:** ![Zorlin](https://avatars.discourse-cdn.com/v4/letter/z/b782af/32.png) [@Zorlin](https://discuss.elastic.co/u/Zorlin)\ **Post date:** [July 19, 2016, 1:32pm UTC](https://discuss.elastic.co/t/lets-encrypt-and-filebeat-tls/55882/1 "2016-07-19T13:32:39Z")
Hey there, I've been following this guide to get an ELK stack up and running - [https://www.digitalocean.com/community/tutorials/how-to-install-elasticsearch-logstash-and-kibana-elk-stack-on-ubuntu-14-04](https://www.digitalocean.com/community/tutorials/how-to-install-elasticsearch-logstash-and-kibana-elk-stack-on-ubuntu-14-04) However, I don't want to set up PKI stuff as defined in the guide (skip to Generate SSL Certificates) - I want to use a public certificate, in this case from Let's Encrypt. They want me to use this command: sudo openssl req -config /etc/ssl/openssl.cnf -x509 -days 3650 -batch -nodes -newkey rsa:2048 -keyout private/logstash-forwarder.key -out certs/logstash-forwarder.crt So instead I used [acme.sh](http://acme.sh) and DNS validation to generate a certificate for my Logstash target - let's call it [logstash.company.com](http://logstash.company.com) - easy, right? Now I have a bunch of certificates: letsencrypt@elk:~/.acme.sh/logstash.company.com$ ls ca.cer logstash.company.com.csr fullchain.cer logstash.company.com.key logstash.company.com.cer logstash.company.com.ssl.conf logstash.company.com.conf In the Logstash config "2-beats-input.conf" (as in the guide) I set ssl\_certificate and ssl\_key to the keys and cert I just generated: ssl\_certificate =\> "/etc/logstash/ssl/logstash.company.com.cer" ssl\_key =\> "/etc/logstash/ssl/logstash.company.com.key" Logstash appears to act like everything is fine. Finally, it came time to set up Filebeat and this is where it fell apart. I copied the cert (logstash.company.com.cer) to another server, which we'll call web1. I configured it as in the guide, and copied the cert to "/etc/pki/tls/certs/logstash-forwarder.crt". I followed the rest of the guide as-is. I thought it was a little odd, however, that it said to set the certificate as "certificate\_authorities: ["/etc/pki/tls/certs/logstash-forwarder.crt"]" Anyways, I think I've tracked down the issue - regardless of whether or not I set certificate\_authorities to the cert, I get this error in Filebeat's logs (I set log level to WARNING): 2016-07-19T13:29:34Z ERR SSL client failed to connect with: x509: certificate signed by unknown authority It's a standard certificate from Let's Encrypt, which ought to be trusted. Maybe I need to copy ca.cer and use that instead? Sorry for the sort of rambling text. I'm just tearing my hair out trying to get this to work. EDIT: I have tried using fullchain.cer instead, but the issue remains - x509: certificate signed by unknown authority. I also double-checked that the cert is using the real CA and not the staging/"fake" CA: Issuer: C=US, O=Let's Encrypt, CN=Let's Encrypt Authority X3 ---
**Author:** ![andrewkroh](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/andrewkroh/32/3784_2.png) [@andrewkroh](https://discuss.elastic.co/u/andrewkroh)\ **Post date:** [July 19, 2016, 1:48pm UTC](https://discuss.elastic.co/t/lets-encrypt-and-filebeat-tls/55882/2 "2016-07-19T13:48:51Z")
> [@Zorlin](#): > > It's a standard certificate from Let's Encrypt, which ought to be trusted. So if your Logstash server is using a certificate that is trusted by the OS then you shouldn't even need to use the `certificate_authorities` option in Filebeat. In theory Filebeat can just use the OS trust store to validate the Let's Encrypt cert. Does `curl -v https://logstash.company.com:5044` say that the certificate is invalid? Or does it say "Empty reply from server" which would mean certificate validation passed. [Secure communication with Logstash | Filebeat Reference [8.11] | Elastic](https://www.elastic.co/guide/en/beats/filebeat/current/configuring-tls-logstash.html#testing-tls-logstash) ---
**Author:** ![andrewkroh](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/andrewkroh/32/3784_2.png) [@andrewkroh](https://discuss.elastic.co/u/andrewkroh)\ **Post date:** [July 19, 2016, 1:54pm UTC](https://discuss.elastic.co/t/lets-encrypt-and-filebeat-tls/55882/3 "2016-07-19T13:54:58Z")
Oh, I just remembered that intermediate CAs are a problem for Logstash at the moment. [https://github.com/logstash-plugins/logstash-input-beats/issues/64](https://github.com/logstash-plugins/logstash-input-beats/issues/64) So you probably need to follow this workaround in Filebeat until it gets fixed: [https://github.com/elastic/beats/issues/1494](https://github.com/elastic/beats/issues/1494) ---
**Author:** ![Zorlin](https://avatars.discourse-cdn.com/v4/letter/z/b782af/32.png) [@Zorlin](https://discuss.elastic.co/u/Zorlin)\ **Post date:** [July 19, 2016, 2:12pm UTC](https://discuss.elastic.co/t/lets-encrypt-and-filebeat-tls/55882/4 "2016-07-19T14:12:56Z")
Hi @andrewkroh, Thanks for taking the time 🙂 Yes, I also tried running without certificate authorities set as in theory it shouldn't be necessary. Same issues occurred. Meanwhile, I tried your curl command on "web1" and got the following output. It's a Debian 8 box. > root@web1:/var/log/mybeat# curl -v [https://logstash.company.com:5044](https://logstash.company.com:5044) > > - Rebuilt URL to: [https://logstash.company.com:5044/](https://logstash.company.com:5044/) > - Hostname was NOT found in DNS cache > - Trying 10.X.Y.Z... > - Connected to [logstash.admin.ecommistry.com](http://logstash.admin.ecommistry.com) (10.X.Y.Z) port 5044 (#0) > - successfully set certificate verify locations: > - CAfile: none > CApath: /etc/ssl/certs > - SSLv3, TLS handshake, Client hello (1): > - SSLv3, TLS handshake, Server hello (2): > - SSLv3, TLS handshake, CERT (11): > - SSLv3, TLS alert, Server hello (2): > - SSL certificate problem: unable to get local issuer certificate > - Closing connection 0 > - SSLv3, TLS alert, Client hello (1): > curl: (60) SSL certificate problem: unable to get local issuer certificate > More details here: [curl - SSL CA Certificates](http://curl.haxx.se/docs/sslcerts.html) Interestingly, I get a totally different result from my OS X laptop: > rain:~ wings$ curl -v [https://logstash.company.com:5044](https://logstash.company.com:5044) > > - Rebuilt URL to: [https://logstash.company.com:5044/](https://logstash.company.com:5044/) > - Trying 10.X.Y.Z... > - Connected to [logstash.company.com](http://logstash.company.com) (10.X.Y.Z) port 5044 (#0) > - TLS 1.2 connection using TLS\_ECDHE\_RSA\_WITH\_AES\_128\_CBC\_SHA256 > - Server certificate: [logstash.company.com](http://logstash.company.com) > - Server certificate: Let's Encrypt Authority X3 > - Server certificate: DST Root CA X3 > > > GET / HTTP/1.1 > > Host: [logstash.company.com:5044](http://logstash.company.com:5044) > > User-Agent: curl/7.43.0 > > Accept: _/_ > > - Empty reply from server > - Connection #0 to host [logstash.company.com](http://logstash.company.com) left intact > curl: (52) Empty reply from server I will look into the intermediate CA issues and the workarounds. Thanks! ---
**Author:** ![Zorlin](https://avatars.discourse-cdn.com/v4/letter/z/b782af/32.png) [@Zorlin](https://discuss.elastic.co/u/Zorlin)\ **Post date:** [July 19, 2016, 2:34pm UTC](https://discuss.elastic.co/t/lets-encrypt-and-filebeat-tls/55882/5 "2016-07-19T14:34:32Z")
Okay, I'm not entirely sure why this works as a fix, but here goes - I simply put ca.cer from the site's Let's Encrypt folder into the configuration: certificate\_authorities: ["/etc/filebeat/ssl/ca.cer"] And suddenly Filebeat works with full TLS. To be clear, ca.cer is (I think?) just the root certificate - and fullchain.cer is the main certificate with the root certificate attached to it. Either that or they are just the intermediate certs and the root cert doesn't come into it at all. Anyways, that seems to be the fix. I might write a blog post in a few days about using LE and Filebeat as they seem to be a good fit. Thank you so much for your help, I've been struggling with this for hours and hadn't thought to try the ca.cer approach. ---
**Author:** ![SockThief](https://avatars.discourse-cdn.com/v4/letter/s/a8b319/32.png) [@SockThief](https://discuss.elastic.co/u/SockThief)\ **Post date:** [July 19, 2016, 11:13pm UTC](https://discuss.elastic.co/t/lets-encrypt-and-filebeat-tls/55882/6 "2016-07-19T23:13:40Z")
Hi zorlin Please write a blog (or at least your steps here) I'm doing exactly the same thing and would like to learn from your mistakes! Did you create the LE cert for the reverse proxy virtual host? ---
**Author:** ![SockThief](https://avatars.discourse-cdn.com/v4/letter/s/a8b319/32.png) [@SockThief](https://discuss.elastic.co/u/SockThief)\ **Post date:** [July 19, 2016, 11:29pm UTC](https://discuss.elastic.co/t/lets-encrypt-and-filebeat-tls/55882/7 "2016-07-19T23:29:18Z")
slightly off topic, I assume that if I had standard self signed certs in place (or even SSL disabled as all components are currently on one machine for testing) it's a simple case to just replace the certs with those from LE? ---
**Author:** ![Zorlin](https://avatars.discourse-cdn.com/v4/letter/z/b782af/32.png) [@Zorlin](https://discuss.elastic.co/u/Zorlin)\ **Post date:** [July 20, 2016, 2:57pm UTC](https://discuss.elastic.co/t/lets-encrypt-and-filebeat-tls/55882/8 "2016-07-20T14:57:35Z")
Hi @SockThief, Will definitely do a writeup at some point soon. No, I created it with two hostnames - "[logs.company.com](http://logs.company.com)" in case I wanted to add SSL to the Kibana (via reverse proxy virtual host) server, and "[logstash.company.com](http://logstash.company.com)" for encrypted Filebeat (and other) traffic. The plan is to have two servers running the full ELK stack, with a shared ElasticSearch cluster. Then loadbalance [logstash.company.com](http://logstash.company.com) via plain DNS round robin, though I may also use the native loadbalancing in Filebeat instead. Then to have an Elastic Load Balancer on AWS in front of the Kibana panel, load balancing between the two Kibana reverse proxies. So, to answer your question - no, I didn't create it for the reverse proxy virtual host, but I did add that as a SAN in case we ever want to change to that setup in the future. For now SSL termination will be done at the ELBs. Now your second question - if I am understanding you correctly, yes - it should be a fairly simple matter to replace the certificates you generate. I can't redo this on the production system, so I'll try to replicate it on my personal server and document the process. ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [August 9, 2016, 1:32pm UTC](https://discuss.elastic.co/t/lets-encrypt-and-filebeat-tls/55882/9 "2016-08-09T13:32:46Z")
This topic was automatically closed after 21 days. New replies are no longer allowed.