Struggling with Kibana to Elasticsearch cluster encryption

[jbrowe]

I've been struggling to get encryption between Kibana and my Elasticsearch cluster working for past two days. My platform is CentOS 7 and I'm using ES version 6.4.0. Here's where I am:

Encryption between ES nodes is working . I used the "elasticsearch-certutil" to create a CA and to create certs for each node per the instructions:

 elasticsearch-certutil ca
 mv elastic-stack-ca.p12 ES-ca.p12
 elasticsearch-certutil cert --ca ES-ca.p12 --name node1 --dns node1.example.com --ip 192.168.10.1
 elasticsearch-certutil cert --ca ES-ca.p12 --name node2 --dns node2.example.com --ip 192.168.10.2
 elasticsearch-certutil cert --ca ES-ca.p12 --name node3 --dns node3.example.com --ip 192.168.10.3
 elasticsearch-certutil cert --ca ES-ca.p12 --name kibana1 --dns kibana1.example.com --ip 192.168.10.4

The "elasticsearch-certutil" produced PKCS12 formatted certificate bundles, which were copied to all the nodes along with the CA cert. Passwords are entered into the elasticsearch keystore. The ES cluster nodes encryption now works fine.

Kibana client communications are successfully SSL/TLS encrypted. I generated the key and csr with openssl and obtained a signed cert from LetsEncrypt. The following configuration is in place:

server.ssl.enabled: true
server.ssl.certificate: /etc/kibana/certs/kibana1_ssl.crt.pem
server.ssl.key: /etc/kibana/private/kibana_ssl.key.pem

The private key password is entered into the Kibana keystore. This all works fine. Note that this private/public key pair were generated by openssl and signed by an external CA, not my elasticsearch CA generated above.

Next I try to secure the communications between kibana1 and the backend ES node1. Communications between node1, node2 and node3 are already successfully encrypted. I just want to add kibana1. Kibana uses pem certs, not PKCS12. The CA and all the ES nodes use PKCS12 bundles, so I extract the pem files from kibana1.p12 using openssl and place them in /etc/kibana/private and /etc/kibana/certs.

openssl pkcs12 -in kibana1.p12 -nocerts -out ./private/kibana1.es_CA.key.pem
openssl pkcs12 -in kibana1.p12 -clcerts -nokeys -out ./certs/kibana1.es_CA.crt.pem

The kibana1.es_CA.key.pem password is entered into the Kibana keystore. My configuration file is modified to include:

elasticsearch.url: "https://node1.example.com:9200" 
elasticsearch.ssl.certificate: /etc/kibana/certs/kibana1.es_CA.crt.pem
elasticsearch.ssl.key: /etc/kibana/private/kibana1.es_CA.key.pem
elasticsearch.ssl.certificateAuthorities: [ "/etc/kibana/certs/ES-ca.p12" ]
elasticsearch.ssl.verificationMode: full

With this configuration, kibana1 can no longer communicate with node1. The error in the kibana.log appears as:

{"type":"log","@timestamp":"2018-11-16T19:37:45Z","tags":["warning","elasticsearch","admin"],"pid":34625,"message":"Unable to revive connection: https://node1.example.com:9200/"}
{"type":"log","@timestamp":"2018-11-16T19:37:45Z","tags":["warning","elasticsearch","admin"],"pid":34625,"message":"No living connections"}

There are many messages like this. I'm not sure how to proceed. Maybe I should ditch the "elasticsearch-certutil" tool and regenerate my certs using openssl from the start. Has anyone done this successfully using a different procedure?

[rashmi (kulkarni)]

@Larry_Gregory can you help here. please ?

Thanks
Rashmi

[Brandon_Kobel]

Hey @jbrowe, can you try using a PEM encoded CA cert for the following kibana.yml setting instead of the PKCS12 certificate: elasticsearch.ssl.certificateAuthorities: [ "/etc/kibana/certs/ES-ca.p12" ]

[jbrowe]

Here's my process for extracting the pem cert:

openssl pkcs12 -in ES-ca.p12 -clcerts -nokeys -out ES-ca.crt.pem

Then I modify the kibana.yml config to:

elasticsearch.ssl.certificateAuthorities: [ "/etc/kibana/certs/ES-ca.crt.pem" ]

No joy. The "unable to revive connection" and "No living connection" error messages seem the same in the kibana logs. If no one has an answer, is there a preferred debug approach? The logs on the elasticsearch server are silent. The kibana logs show connection failures but no cause.

[Brandon_Kobel]

If you change elasticsearch.ssl.verificationMode: full in your kibana.yml to elasticsearch.ssl.verificationMode: certificate does it work?

You'll also want to ensure that the ES-ca.crt.pem that you're using for the elasticsearch.ssl.certificateAuthorities is valid, and you can do so by using a command similar to the following to connect to Elasticsearch, substituting your own filepath, URL and username/passwords:

curl --cacert /path/to/ES-ca.crt.pem http://localhost:9200/ -u elastic:changeme

[jbrowe]

Changing to elasticsearch.ssl.verificationMode: certificate results in the same failure to connect.

Now I'm worried that I've misunderstood. All my elasticsearch nodes have the following in their elasticsearch.yml configs:

# ---------------------------------- X-Pack ------------------------------------
#
xpack.security.enabled: true
xpack.security.transport.ssl.enabled: true
xpack.security.transport.ssl.verification_mode: full 
xpack.security.transport.ssl.keystore.path: certs/node1.p12 
xpack.security.transport.ssl.truststore.path: certs/node1.p12
xpack.monitoring.collection.enabled: true

I assumed this is all that is needed to guarantee that ES node-to-node encrypted communications was enabled. Even though SSL is enabled on my ES nodes, my Kibana node is still able to connect without configuring Kibana-to-ES SSL encryption (using http:// instead of https://). Is this correct behavior or am I not actually using encryption between ES nodes on my cluster?

I followed your recommendation and...

curl --cacert /path/to/ES-ca.crt.pem http://localhost:9200/ -u elastic:changeme

...works fine. However, if I replace http:// with https:// in verbose mode, i.e.:

curl --cacert /path/to/ES-ca.crt.pem https://localhost:9200/ -u elastic:changeme -v

Then I get the error:

* About to connect() to node1.example.com port 9200 (#0)
*   Trying 192.168.10.1....
* Connected to node1.example.com (192.168.10.1) port 9200 (#0)
* Initializing NSS with certpath: sql:/etc/pki/nssdb
*   CAfile: ES-ca.crt.pem
  CApath: none
* NSS error -12263 (SSL_ERROR_RX_RECORD_TOO_LONG)
* SSL received a record that exceeded the maximum permissible length.
* Closing connection 0
curl: (35) SSL received a record that exceeded the maximum permissible length.

I extracted the CA cert from node.p12 and verified that it is indeed the same as the CA cert ES-ca.crt.pem.

[Brandon_Kobel]

Changing to elasticsearch.ssl.verificationMode: certificate results in the same failure to connect.

Apologies for not being more clear earlier. I was attempting to see if changing elasticsearch.ssl.verificationMode: certificate for debugging purposes, and it's completely fine to set this to full. When set to full we perform hostname validation on the certificate, so I was trying to see if the hostnames for the certs was the cause of the problem.

I assumed this is all that is needed to guarantee that ES node-to-node encrypted communications was enabled. Even though SSL is enabled on my ES nodes, my Kibana node is still able to connect without configuring Kibana-to-ES SSL encryption (using http:// instead of https:// ). Is this correct behavior or am I not actually using encryption between ES nodes on my cluster?

Using these settings, you have encrypted the communication between ES nodes in your cluster. However, you haven't enabled TLS for your http based communication with Elasticsearch. To do that, you'll want to set xpack.security.http.ssl.enabled: true in your elasticsearch.yml. Once you have done so, you should see the curl --cacert /path/to/ES-ca.crt.pem https://localhost:9200/ -u elastic:changeme -v command succeed.

[jbrowe]

Thank you very much. That was what I missed. Here is what I finally added to my elasticsearch.yml on each cluster node:

# Encryption of https is configured differently and needs the following
xpack.security.http.ssl.enabled: true
xpack.security.http.ssl.keystore.path: certs/node1.p12
xpack.security.http.ssl.truststore.path: certs/node1.p12

Then I added the private key password to the keystore on each node:

$ES_HOME/bin/elasticsearch-keystore add xpack.security.http.ssl.keystore.secure_password
$ES_HOME/bin/elasticsearch-keystore add xpack.security.http.ssl.truststore.secure_password

Now encryption works from Kibana to Elasticsearch nodes. Next I need to tackle the Logstash-to-Kibana and Logstash monitoring and management encryption. Thanks again.