# Struggling with Kibana to Elasticsearch cluster encryption **URL:** \ **Category:** Kibana\ **Tags:** elastic-stack-security\ **Created:** [November 16, 2018, 10:06pm UTC](https://discuss.elastic.co/t/struggling-with-kibana-to-elasticsearch-cluster-encryption/157140 "2018-11-16T22:06:51Z")\ **Posts on this page:** 9\ **Page:** 1
**Author:** ![jbrowe](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jbrowe/32/99821_2.png) [@jbrowe](https://discuss.elastic.co/u/jbrowe)\ **Post date:** [November 16, 2018, 10:06pm UTC](https://discuss.elastic.co/t/struggling-with-kibana-to-elasticsearch-cluster-encryption/157140/1 "2018-11-16T22:06:51Z")
I've been struggling to get encryption between Kibana and my Elasticsearch cluster working for past two days. My platform is CentOS 7 and I'm using ES version 6.4.0. Here's where I am: Encryption between ES nodes is working . I used the "elasticsearch-certutil" to create a CA and to create certs for each node per the instructions: ``` elasticsearch-certutil ca mv elastic-stack-ca.p12 ES-ca.p12 elasticsearch-certutil cert --ca ES-ca.p12 --name node1 --dns node1.example.com --ip 192.168.10.1 elasticsearch-certutil cert --ca ES-ca.p12 --name node2 --dns node2.example.com --ip 192.168.10.2 elasticsearch-certutil cert --ca ES-ca.p12 --name node3 --dns node3.example.com --ip 192.168.10.3 elasticsearch-certutil cert --ca ES-ca.p12 --name kibana1 --dns kibana1.example.com --ip 192.168.10.4 ``` The "elasticsearch-certutil" produced PKCS12 formatted certificate bundles, which were copied to all the nodes along with the CA cert. Passwords are entered into the elasticsearch keystore. The ES cluster nodes encryption now works fine. Kibana client communications are successfully SSL/TLS encrypted. I generated the key and csr with openssl and obtained a signed cert from LetsEncrypt. The following configuration is in place: ``` server.ssl.enabled: true server.ssl.certificate: /etc/kibana/certs/kibana1_ssl.crt.pem server.ssl.key: /etc/kibana/private/kibana_ssl.key.pem ``` The private key password is entered into the Kibana keystore. This all works fine. Note that this private/public key pair were generated by openssl and signed by an external CA, not my elasticsearch CA generated above. Next I try to secure the communications between kibana1 and the backend ES node1. Communications between node1, node2 and node3 are already successfully encrypted. I just want to add kibana1. Kibana uses pem certs, not PKCS12. The CA and all the ES nodes use PKCS12 bundles, so I extract the pem files from kibana1.p12 using openssl and place them in /etc/kibana/private and /etc/kibana/certs. ``` openssl pkcs12 -in kibana1.p12 -nocerts -out ./private/kibana1.es_CA.key.pem openssl pkcs12 -in kibana1.p12 -clcerts -nokeys -out ./certs/kibana1.es_CA.crt.pem ``` The kibana1.es\_CA.key.pem password is entered into the Kibana keystore. My configuration file is modified to include: ``` elasticsearch.url: "https://node1.example.com:9200" elasticsearch.ssl.certificate: /etc/kibana/certs/kibana1.es_CA.crt.pem elasticsearch.ssl.key: /etc/kibana/private/kibana1.es_CA.key.pem elasticsearch.ssl.certificateAuthorities: ["/etc/kibana/certs/ES-ca.p12"] elasticsearch.ssl.verificationMode: full ``` With this configuration, kibana1 can no longer communicate with node1. The error in the kibana.log appears as: ``` {"type":"log","@timestamp":"2018-11-16T19:37:45Z","tags":["warning","elasticsearch","admin"],"pid":34625,"message":"Unable to revive connection: https://node1.example.com:9200/"} {"type":"log","@timestamp":"2018-11-16T19:37:45Z","tags":["warning","elasticsearch","admin"],"pid":34625,"message":"No living connections"} ``` There are many messages like this. I'm not sure how to proceed. Maybe I should ditch the "elasticsearch-certutil" tool and regenerate my certs using openssl from the start. Has anyone done this successfully using a different procedure? ---
**Author:** ![rashmi](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/rashmi/32/16391_2.png) [@rashmi](https://discuss.elastic.co/u/rashmi)\ **Post date:** [November 16, 2018, 10:38pm UTC](https://discuss.elastic.co/t/struggling-with-kibana-to-elasticsearch-cluster-encryption/157140/2 "2018-11-16T22:38:04Z")
@Larry_Gregory can you help here. please ? Thanks Rashmi ---
**Author:** ![Brandon\_Kobel](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/brandon_kobel/32/14829_2.png) [@Brandon\_Kobel](https://discuss.elastic.co/u/Brandon_Kobel)\ **Post date:** [November 19, 2018, 3:41pm UTC](https://discuss.elastic.co/t/struggling-with-kibana-to-elasticsearch-cluster-encryption/157140/3 "2018-11-19T15:41:58Z")
Hey @jbrowe, can you try using a PEM encoded CA cert for the following kibana.yml setting instead of the PKCS12 certificate: `elasticsearch.ssl.certificateAuthorities: ["/etc/kibana/certs/ES-ca.p12"]` ---
**Author:** ![jbrowe](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jbrowe/32/99821_2.png) [@jbrowe](https://discuss.elastic.co/u/jbrowe)\ **Post date:** [November 19, 2018, 10:55pm UTC](https://discuss.elastic.co/t/struggling-with-kibana-to-elasticsearch-cluster-encryption/157140/4 "2018-11-19T22:55:05Z")
Here's my process for extracting the pem cert: ``` openssl pkcs12 -in ES-ca.p12 -clcerts -nokeys -out ES-ca.crt.pem ``` Then I modify the kibana.yml config to: ``` elasticsearch.ssl.certificateAuthorities: ["/etc/kibana/certs/ES-ca.crt.pem"] ``` No joy. The "unable to revive connection" and "No living connection" error messages seem the same in the kibana logs. If no one has an answer, is there a preferred debug approach? The logs on the elasticsearch server are silent. The kibana logs show connection failures but no cause. ---
**Author:** ![Brandon\_Kobel](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/brandon_kobel/32/14829_2.png) [@Brandon\_Kobel](https://discuss.elastic.co/u/Brandon_Kobel)\ **Post date:** [November 20, 2018, 3:08pm UTC](https://discuss.elastic.co/t/struggling-with-kibana-to-elasticsearch-cluster-encryption/157140/5 "2018-11-20T15:08:55Z")
If you change `elasticsearch.ssl.verificationMode: full` in your kibana.yml to `elasticsearch.ssl.verificationMode: certificate` does it work? You'll also want to ensure that the `ES-ca.crt.pem` that you're using for the `elasticsearch.ssl.certificateAuthorities` is valid, and you can do so by using a command similar to the following to connect to Elasticsearch, substituting your own filepath, URL and username/passwords: ```auto curl --cacert /path/to/ES-ca.crt.pem http://localhost:9200/ -u elastic:changeme ``` ---
**Author:** ![jbrowe](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jbrowe/32/99821_2.png) [@jbrowe](https://discuss.elastic.co/u/jbrowe)\ **Post date:** [November 20, 2018, 10:07pm UTC](https://discuss.elastic.co/t/struggling-with-kibana-to-elasticsearch-cluster-encryption/157140/6 "2018-11-20T22:07:21Z")
Changing to `elasticsearch.ssl.verificationMode: certificate` results in the same failure to connect. Now I'm worried that I've misunderstood. All my elasticsearch nodes have the following in their `elasticsearch.yml` configs: ``` # ---------------------------------- X-Pack ------------------------------------ # xpack.security.enabled: true xpack.security.transport.ssl.enabled: true xpack.security.transport.ssl.verification_mode: full xpack.security.transport.ssl.keystore.path: certs/node1.p12 xpack.security.transport.ssl.truststore.path: certs/node1.p12 xpack.monitoring.collection.enabled: true ``` I assumed this is all that is needed to guarantee that ES node-to-node encrypted communications was enabled. Even though SSL is enabled on my ES nodes, my Kibana node is still able to connect without configuring Kibana-to-ES SSL encryption (using `http://` instead of `https://`). Is this correct behavior or am I not actually using encryption between ES nodes on my cluster? I followed your recommendation and... ``` curl --cacert /path/to/ES-ca.crt.pem http://localhost:9200/ -u elastic:changeme ``` ...works fine. However, if I replace `http://` with `https://` in verbose mode, i.e.: ``` curl --cacert /path/to/ES-ca.crt.pem https://localhost:9200/ -u elastic:changeme -v ``` Then I get the error: ``` * About to connect() to node1.example.com port 9200 (#0) * Trying 192.168.10.1.... * Connected to node1.example.com (192.168.10.1) port 9200 (#0) * Initializing NSS with certpath: sql:/etc/pki/nssdb * CAfile: ES-ca.crt.pem CApath: none * NSS error -12263 (SSL_ERROR_RX_RECORD_TOO_LONG) * SSL received a record that exceeded the maximum permissible length. * Closing connection 0 curl: (35) SSL received a record that exceeded the maximum permissible length. ``` I extracted the CA cert from node.p12 and verified that it is indeed the same as the CA cert ES-ca.crt.pem. ---
**Author:** ![Brandon\_Kobel](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/brandon_kobel/32/14829_2.png) [@Brandon\_Kobel](https://discuss.elastic.co/u/Brandon_Kobel)\ **Post date:** [November 20, 2018, 10:22pm UTC](https://discuss.elastic.co/t/struggling-with-kibana-to-elasticsearch-cluster-encryption/157140/7 "2018-11-20T22:22:53Z")
> Changing to `elasticsearch.ssl.verificationMode: certificate` results in the same failure to connect. Apologies for not being more clear earlier. I was attempting to see if changing `elasticsearch.ssl.verificationMode: certificate` for debugging purposes, and it's completely fine to set this to `full`. When set to `full` we perform hostname validation on the certificate, so I was trying to see if the hostnames for the certs was the cause of the problem. > I assumed this is all that is needed to guarantee that ES node-to-node encrypted communications was enabled. Even though SSL is enabled on my ES nodes, my Kibana node is still able to connect without configuring Kibana-to-ES SSL encryption (using `http://` instead of `https://` ). Is this correct behavior or am I not actually using encryption between ES nodes on my cluster? Using these settings, you have encrypted the communication between ES nodes in your cluster. However, you haven't enabled TLS for your http based communication with Elasticsearch. To do that, you'll want to set `xpack.security.http.ssl.enabled: true` in your `elasticsearch.yml`. Once you have done so, you should see the `curl --cacert /path/to/ES-ca.crt.pem https://localhost:9200/ -u elastic:changeme -v` command succeed. ---
**Author:** ![jbrowe](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jbrowe/32/99821_2.png) [@jbrowe](https://discuss.elastic.co/u/jbrowe)\ **Post date:** [November 21, 2018, 1:16am UTC](https://discuss.elastic.co/t/struggling-with-kibana-to-elasticsearch-cluster-encryption/157140/8 "2018-11-21T01:16:12Z")
Thank you very much. That was what I missed. Here is what I finally added to my `elasticsearch.yml` on each cluster node: ``` # Encryption of https is configured differently and needs the following xpack.security.http.ssl.enabled: true xpack.security.http.ssl.keystore.path: certs/node1.p12 xpack.security.http.ssl.truststore.path: certs/node1.p12 ``` Then I added the private key password to the keystore on each node: ``` $ES_HOME/bin/elasticsearch-keystore add xpack.security.http.ssl.keystore.secure_password $ES_HOME/bin/elasticsearch-keystore add xpack.security.http.ssl.truststore.secure_password ``` Now encryption works from Kibana to Elasticsearch nodes. Next I need to tackle the Logstash-to-Kibana and Logstash monitoring and management encryption. Thanks again. ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [December 19, 2018, 1:16am UTC](https://discuss.elastic.co/t/struggling-with-kibana-to-elasticsearch-cluster-encryption/157140/9 "2018-12-19T01:16:16Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.