# Ssl setup for filebeat and logstash in docker getting desperate now **URL:** \ **Category:** Beats\ **Tags:** docker, filebeat\ **Created:** [August 18, 2024, 9:34am UTC](https://discuss.elastic.co/t/ssl-setup-for-filebeat-and-logstash-in-docker-getting-desperate-now/365078 "2024-08-18T09:34:57Z")\ **Posts on this page:** 13\ **Page:** 1
**Author:** ![daverodgers](https://avatars.discourse-cdn.com/v4/letter/d/779978/32.png) [@daverodgers](https://discuss.elastic.co/u/daverodgers)\ **Post date:** [August 18, 2024, 9:34am UTC](https://discuss.elastic.co/t/ssl-setup-for-filebeat-and-logstash-in-docker-getting-desperate-now/365078/1 "2024-08-18T09:34:57Z")
Morning everyone. im getting desperate now so i really need your help please. I cannot find a guide anywhere that is for a production setup , where you have nodes and beats running on different servers, it all seems to be locahost in examples, and i cant seem to make the successful jump to setup my cluster correctly. Ive been battling for over a week trying to get a working SSL setup in my Elastic stack but i cant seem to get a working combination of certs created. My Linux based cluster is this. (ive changed some details slightly for security/common sense) ES, kibana and Logstash running on a single host (hosted in linode) outside of our internal network where our beats will be running. external ip = 139.100.100.100 dns host name = [monitor.myelastic.com](http://monitor.myelastic.com) server name = [flanders.mydomain.com](http://flanders.mydomain.com) Firewalled off so it isnt accessible by anyone outside our network. Logstash running on default port 5044. Filebeat running on any of our internal linux servers to send logs to logstash. This traffic is the only traffic going over the internet so we need SSL enabled. I want to use elastic cert util to create the certs that will be used by filebeat to send log data to logstash with SSL enabled. I know best practice in some guides ive read is to use a 3rd party cert like lets encrypt but i was hoping using the inbuilt util it would be easier and its sufficient for us. Im trying to come up with a valid instances.yml file to use with the util but im not sure what name, ip and dns details to use. Ive tied a few but nothing seems to work. With the details ive provided could someone suggest a correct instance.yml i can use? and also the correct filebeat.yml and maybe logstash.conf SSL lines so that they both talk to each other using these new certs. elastic community...you are my only hope. ---
**Author:** ![stephenb](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stephenb/32/40856_2.png) [@stephenb](https://discuss.elastic.co/u/stephenb)\ **Post date:** [August 18, 2024, 2:44pm UTC](https://discuss.elastic.co/t/ssl-setup-for-filebeat-and-logstash-in-docker-getting-desperate-now/365078/2 "2024-08-18T14:44:34Z")
Hi @daverodgers Did you look at the example instance file [here](https://www.elastic.co/guide/en/elasticsearch/reference/current/certutil.html#certutil-silent) Should be straight forward share your anonymized one... For the https connection... That's how it usually works you show us :slight\_smile And show the command you ran to generate Then test with curl to connect to elasticsearch? Does logstash already connect to elasticsearch since on same server? Do you need logstash? Logstash takes a bit more.. What version are you running? Let's get curl over ssl working first...then move on to logstash. Also, did you make sure you bound elasticsearch to the network? You did not share your docker setup... So it's hard for us to help with specifics.... You told us your journey but did not provide much in the way of configs ---
**Author:** ![daverodgers](https://avatars.discourse-cdn.com/v4/letter/d/779978/32.png) [@daverodgers](https://discuss.elastic.co/u/daverodgers)\ **Post date:** [August 18, 2024, 8:11pm UTC](https://discuss.elastic.co/t/ssl-setup-for-filebeat-and-logstash-in-docker-getting-desperate-now/365078/3 "2024-08-18T20:11:47Z")
hi Stephen thanks for taking the time to reply. i'd like to attempt another go at creating my instances file. but if i may ask a question. If ES, logstash and kibana all exist on the same host, does the instances file only need one "name" entry under the instances section? So in my case it would be ```auto instances: - name: flanders ip: - 139.100.100.100 dns: - monitor.myelastic.com - flanders.mydomain.com ``` Upto now i have been creating a "name" for all 3 apps but maybe ive been thinking about this all wrong. I will try and answer some of your other questions: Does logstash already connect to elasticsearch since on same server? - yes logstash is connecting to ES Do you need logstash? - yes we want to manipulate some of the data before it goes to ES. What version are you running? - latest version (as this is a brand new install) I will try creating a cert based on the above example instances i gave until i get your response to that , and see if its works. Just for info, this is purely a cert/ssl issue. If i disable SSL in my logstash conf file (input section) the data flows from filebeat, through LS and into ES perfectly fine. It only stops working when i try to enable SSL. thanks ---
**Author:** ![leandrojmp](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/leandrojmp/32/107231_2.png) [@leandrojmp](https://discuss.elastic.co/u/leandrojmp)\ **Post date:** [August 18, 2024, 9:40pm UTC](https://discuss.elastic.co/t/ssl-setup-for-filebeat-and-logstash-in-docker-getting-desperate-now/365078/4 "2024-08-18T21:40:23Z")
> [@daverodgers](#): > > If ES, logstash and kibana all exist on the same host, does the instances file only need one "name" entry under the instances section? You need to add each dns that you are going to use, even if they are on the same server. If you are going to access using `dns1.domain` and `dns2.domain`, then both need to be added to the certificate. > [@daverodgers](#): > > Just for info, this is purely a cert/ssl issue. If i disable SSL in my logstash conf file (input section) the data flows from filebeat, through LS and into ES perfectly fine. It only stops working when i try to enable SSL. You need to provide context, share your logstash configuration, share your filebeat configuration, share the log errors that you are getting. If things stops working when you enable SSL then your configuration is wrong or your certificate is wrong, but you need to share the errors you are getting and the configurations you are using. Without this is pretty hard to troubleshoot. ---
**Author:** ![daverodgers](https://avatars.discourse-cdn.com/v4/letter/d/779978/32.png) [@daverodgers](https://discuss.elastic.co/u/daverodgers)\ **Post date:** [August 19, 2024, 11:43am UTC](https://discuss.elastic.co/t/ssl-setup-for-filebeat-and-logstash-in-docker-getting-desperate-now/365078/5 "2024-08-19T11:43:06Z")
logstash config ```auto input { beats { port => 5044 ssl_enabled => true ssl_certificate_authorities => "certs/ca/ca.crt" ssl_certificate => "certs/newcerts/logstash/logstash.crt" ssl_key => "certs/ca/ca.key" } } output { elasticsearch { hosts => ["https://es01:9200"] user => "elastic" password => "password123" ssl_enabled => true ssl_certificate_authorities => "certs/ca/ca.crt" } } ``` the filebeat.yml is ```auto filebeat.inputs: - type: log paths: - /var/log/*.log - /var/log/messages output.logstash: hosts: ["logstashserver-external-ip:5044"] ssl.enabled: true ssl.certificate_authorities: "certs/ca/ca.crt" ssl.certificate: "certs/ca/ca.crt" ssl.key: "certs/ca/ca.key" ``` if i try a test curl command from inside the filebeat container ```auto curl -v -k --cacert ./certs/ca/ca.crt https://logstashserver-external-ip:5044 ``` i get the following response ```auto * successfully set certificate verify locations: * CAfile: ./certs/ca/ca.crt CApath: /etc/ssl/certs * TLSv1.3 (OUT), TLS handshake, Client hello (1): * TLSv1.3 (IN), TLS handshake, Server hello (2): * TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8): * TLSv1.3 (IN), TLS handshake, Request CERT (13): * TLSv1.3 (IN), TLS handshake, Certificate (11): * TLSv1.3 (IN), TLS handshake, CERT verify (15): * TLSv1.3 (OUT), TLS alert, decrypt error (563): * error:0407E085:rsa routines:RSA_verify_PKCS1_PSS_mgf1:first octet invalid * Closing connection 0 curl: (35) error:0407E085:rsa routines:RSA_verify_PKCS1_PSS_mgf1:first octet invalid ``` and in my filebeat container logs i see this error: ```auto Failed to connect to backoff(async(tcp://logstash-external-ip:5044)): tls: invalid signature by the server certificate: crypto/rsa: verification error","service.name":"filebeat","ecs.version":"1.6.0"} ``` thanks ---
**Author:** ![leandrojmp](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/leandrojmp/32/107231_2.png) [@leandrojmp](https://discuss.elastic.co/u/leandrojmp)\ **Post date:** [August 19, 2024, 1:08pm UTC](https://discuss.elastic.co/t/ssl-setup-for-filebeat-and-logstash-in-docker-getting-desperate-now/365078/6 "2024-08-19T13:08:26Z")
Both the response of your curl and the log from beats suggests that there is something wrong with your certificate. Also, you should not use `-k` when validate the certificate with the `curl` command. How did you create the certificate and key for the beats input? Is the key in the pkcs8 format? It is required. For some reason the Beats documentation does not have any example on how to generate the keys and certificates, but you can follow the Elastic Agent documentation on how to generate those keys and certificates, it is basically the same thing. You can check the documentation [here](https://www.elastic.co/guide/en/fleet/current/secure-logstash-connections.html). Basically you create a CA, then using this CA you create a **client** certificate in the _pem_ format, on this certificate you do not specify any dns or ip address. After that you create the **server** certificate, in this certificate you specify the dns and ip address that are valid, it is the one you create using the `instances.yml` file. Then you convert the key to the pkcs8 format as it is required. ---
**Author:** ![daverodgers](https://avatars.discourse-cdn.com/v4/letter/d/779978/32.png) [@daverodgers](https://discuss.elastic.co/u/daverodgers)\ **Post date:** [August 19, 2024, 1:25pm UTC](https://discuss.elastic.co/t/ssl-setup-for-filebeat-and-logstash-in-docker-getting-desperate-now/365078/7 "2024-08-19T13:25:59Z")
Hi leandro I created the certs following the elastic guide on how to setup elk using docker compose. so it auto creates them in a container for me based on what is in my compose file. so the key is n whatever format the elastic util creates by default. this is the line in the elastic guide/docker compose file that creates it: ```auto bin/elasticsearch-certutil cert --silent --pem -out config/certs/certs.zip --in config/certs/instances.yml --ca-cert config/certs/ca/ca.crt --ca-key config/certs/ca/ca.key; ``` i guess i will try and recreate my certs again and reset, ive changed so much trying to resolve this that it is very confusing. ---
**Author:** ![leandrojmp](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/leandrojmp/32/107231_2.png) [@leandrojmp](https://discuss.elastic.co/u/leandrojmp)\ **Post date:** [August 19, 2024, 2:14pm UTC](https://discuss.elastic.co/t/ssl-setup-for-filebeat-and-logstash-in-docker-getting-desperate-now/365078/8 "2024-08-19T14:14:59Z")
The key needs to ben the pkcs8 format, it will not work if it is not in this format. The elasticsearch-certutil tool does not create keys in this format, you need to convert using openssl as explained in the documentation linked. ---
**Author:** ![daverodgers](https://avatars.discourse-cdn.com/v4/letter/d/779978/32.png) [@daverodgers](https://discuss.elastic.co/u/daverodgers)\ **Post date:** [August 19, 2024, 3:09pm UTC](https://discuss.elastic.co/t/ssl-setup-for-filebeat-and-logstash-in-docker-getting-desperate-now/365078/9 "2024-08-19T15:09:55Z")
hi thanks for that info. just to confirm, are we talking about the ca.key that is created? or the other .key files that get created in the ES / logstash and kibana folders? and once converted, i presume i then need to manually copy them to the cert volume location so the containers can read them, and change any cfg file to reference the new keys (if the name changes)? thanks again. ---
**Author:** ![daverodgers](https://avatars.discourse-cdn.com/v4/letter/d/779978/32.png) [@daverodgers](https://discuss.elastic.co/u/daverodgers)\ **Post date:** [August 20, 2024, 9:13am UTC](https://discuss.elastic.co/t/ssl-setup-for-filebeat-and-logstash-in-docker-getting-desperate-now/365078/10 "2024-08-20T09:13:46Z")
ok so i have created the pkcs8 key using the openssl command. i have changed my filebeat.yml file to reference the new version of the key: ```auto output.logstash: hosts: elk-logstash01-1:5044 index: filebeat ssl.certificate_authorities: ["certs2/ca/ca.crt"] ssl.verification_mode: none ssl.certificate: "certs2/logstash/logstash.crt" ssl.key: "certs2/logstash/logstash.pkcs8.key" ``` in the filebeat logs i still see this: ```auto "message":"Failed to connect to backoff(async(tcp://elk-logstash01-1:5044)): tls: invalid signature by the server certificate: crypto/rsa: verification error","service.name":"filebeat","ecs.version":"1.6.0"} and Attempting to reconnect to backoff(async(tcp://elk-logstash01-1:5044)) with 3 reconnect attempt ``` ---
**Author:** ![daverodgers](https://avatars.discourse-cdn.com/v4/letter/d/779978/32.png) [@daverodgers](https://discuss.elastic.co/u/daverodgers)\ **Post date:** [August 20, 2024, 9:35am UTC](https://discuss.elastic.co/t/ssl-setup-for-filebeat-and-logstash-in-docker-getting-desperate-now/365078/11 "2024-08-20T09:35:39Z")
stephen when i run curl -v --cacert /var/lib/docker/volumes/elk\_certs/\_data/ca/ca.crt [https://made-up-hostname.com:5044](https://made-up-hostname.com:5044) i get this response: ```auto * Connected to made-up-hostname.com (etxernal-server-ip) port 5044 (#0) * ALPN: offers h2,http/1.1 * TLSv1.3 (OUT), TLS handshake, Client hello (1): * CAfile: /var/lib/docker/volumes/elk_certs/_data/ca/ca.crt * CApath: /etc/ssl/certs * TLSv1.3 (IN), TLS handshake, Server hello (2): * TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8): * TLSv1.3 (IN), TLS handshake, Request CERT (13): * TLSv1.3 (IN), TLS handshake, Certificate (11): * TLSv1.3 (IN), TLS handshake, CERT verify (15): * TLSv1.3 (IN), TLS handshake, Finished (20): * TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1): * TLSv1.3 (OUT), TLS handshake, Certificate (11): * TLSv1.3 (OUT), TLS handshake, Finished (20): * SSL connection using TLSv1.3 / TLS_AES_256_GCM_SHA384 * ALPN: server did not agree on a protocol. Uses default. * Server certificate: * subject: CN=logstash * start date: Aug 19 21:06:50 2024 GMT * expire date: Aug 19 21:06:50 2027 GMT * subjectAltName: host "made-up-hostname.com" matched cert's "made-up-hostname.com" * issuer: CN=Elastic Certificate Tool Autogenerated CA * SSL certificate verify ok. * using HTTP/1.x > GET / HTTP/1.1 > Host: made-up-hostname:5044 > User-Agent: curl/7.88.1 > Accept: */* > * TLSv1.3 (IN), TLS alert, bad certificate (554): * OpenSSL SSL_read: OpenSSL/3.0.13: error:0A000412:SSL routines::sslv3 alert bad certificate, errno 0 * Closing connection 0 curl: (56) OpenSSL SSL_read: OpenSSL/3.0.13: error:0A000412:SSL routines::sslv3 alert bad certificate, errno 0 ``` the bad cert error obviously matches the error i see in the logstash logs. so i think my problem is the CA cert that was created has an issue ☹ but I created it using this command inside the docker compose file: ```auto bin/elasticsearch-certutil ca --silent --pem -out config/certs2/ca.zip; unzip config/certs2/ca.zip -d config/certs2 ``` any ideas? ---
**Author:** ![daverodgers](https://avatars.discourse-cdn.com/v4/letter/d/779978/32.png) [@daverodgers](https://discuss.elastic.co/u/daverodgers)\ **Post date:** [August 20, 2024, 11:36am UTC](https://discuss.elastic.co/t/ssl-setup-for-filebeat-and-logstash-in-docker-getting-desperate-now/365078/12 "2024-08-20T11:36:06Z")
> [@daverodgers](#): > > `TLSv1.3 (IN), TLS alert, bad certificate (554)` I think i have fixed this, so dont spend much time replying (if anyone was planning to do so). I will update if i have indeed resolved it incase it helps anyone else. ---
**Author:** ![daverodgers](https://avatars.discourse-cdn.com/v4/letter/d/779978/32.png) [@daverodgers](https://discuss.elastic.co/u/daverodgers)\ **Post date:** [August 21, 2024, 10:30am UTC](https://discuss.elastic.co/t/ssl-setup-for-filebeat-and-logstash-in-docker-getting-desperate-now/365078/13 "2024-08-21T10:30:08Z")
to resolve this i recreated all the certs for CA and nodes using these commands. note these were inside my docker compose file, in a setup container that creates the certs first before starting up the ELK related containers. The docker compose creates a docker volume called certs where the new certs were copied to. I also used an openssl command to convert the .key file that the elasticsearch util creates into the pkcs8 format, which seemed to be the last bit of the puzzle. (this is in a guide linked earlier in this thread) ```auto bin/elasticsearch-certutil ca --silent --pem -out config/certs2/ca.zip; unzip config/certs2/ca.zip -d config/certs2; bin/elasticsearch-certutil cert --silent --pem -out config/certs2/certs.zip --in config/certs2/instances.yml --ca-cert config/certs2/ca/ca.crt --ca-key config/certs2/ca/ca.key; unzip config/certs2/certs.zip -d config/certs2; ``` My filebeat.yml file looks like this: ```auto filebeat.inputs: - type: filestream id: default-filestream paths: - ingest_data/*.log - /var/log/*.log output.logstash: hosts: elk-logstash01-1:5044 index: filebeat ssl.certificate_authorities: ["config/certs2/ca/ca.crt"] ssl.verification_mode: none ssl.certificate: "config/certs2/logstash/logstash.crt" ssl.key: "config/certs2/logstash/logstash.pkcs8.key" ``` my logstash.conf file is this: ```auto Input { beats { port => 5044 ssl_enabled => true ssl_certificate_authorities => "config/certs2/ca/ca.crt" ssl_certificate => "config/certs2/logstash/logstash.crt" ssl_key => "config/certs2/logstash/logstash.pkcs8.key" } } output { elasticsearch { hosts => ["https://es01:9200"] user => "elastic" password => " ****" ssl_enabled => true ssl_certificate_authorities => "config/certs2/ca/ca.crt" } } ``` thanks to everyone who replied.