Node to node encryption works, but NOT the HTTP client one

[mlemartien (Jacques Clement)]

Hi there,

I have been trying to encrypt communication between the cluster nodes (this works like a charm), the browser and Kibana (works well too) and finally for the HTTP Client traffic. This latter does NOT work.

What I did so far:

• Generated one PKCS12 certificate per node (3 in totals, 1 of them being a coordinating node only and also running Kibana)
• Encryption works well with those certificates with the verification mode to full
• Configured kibana.yml to make it point to https://my_coordinating_node

I keep getting those error messages:
{"type":"log","@timestamp":"2018-04-27T15:10:25Z","tags":["warning","elasticsearch","admin"],"pid":28466,"message":"Unable to revive connection: https://172.28.128.21:9200/"}
{"type":"log","@timestamp":"2018-04-27T15:10:25Z","tags":["warning","elasticsearch","admin"],"pid":28466,"message":"No living connections"}

In the kibana.stdout log file but the kibana.stderr file remains empty.

I have a cluster of 3 nodes: 1 coordinating-only + kibana, 2 master/data

I tried so many variations, I am going crazy. Any ideas or leads would be more than welcome!

[ My elasticsearch.yml files ]

cluster.name: eLABsticsearch
node.name: client01 ( or elastic01 or elastic02 )
node.data: false ( or true for elastic01 and elastic02 )
node.master: false ( or true for elastic01 and elastic02 )
node.ingest: false
node.ml: false
search.remote.connect: false

path.data: /var/lib/elasticsearch
path.logs: /var/log/elasticsearch

network.host: 172.28.128.21 ( or 172.28.128.11 or 172.28.128.12 )
http.port: 9200

discovery.zen.ping.unicast.hosts: ["172.28.128.11","172.28.128.12"]
discovery.zen.minimum_master_nodes: 1

xpack.security.transport.ssl.verification_mode: full
xpack.security.transport.ssl.keystore.path: /etc/elasticsearch/${node.name}.p12
xpack.security.transport.ssl.truststore.path: /etc/elasticsearch/${node.name}.p12

xpack.security.http.ssl.enabled: true
xpack.security.http.ssl.keystore.path: /etc/elasticsearch/${node.name}.p12
xpack.security.http.ssl.truststore.path: /etc/elasticsearch/${node.name}.p12

[ My kibana.yml file ]

server.host: "172.28.128.21"
elasticsearch.url: "https://172.28.128.21:9200"
elasticsearch.username: "kibana"
elasticsearch.password: "********"
server.ssl.enabled: true
server.ssl.certificate: /etc/kibana/client01.crt
server.ssl.key: /etc/kibana/client01.key

[ikakavas (Ioannis Kakavas)]

You need to configure Kibana so that it can trust the certificate that Elasticsearch presents it when attempting to connect via http over TLS.
The p12 keystores that you have created with certutil ( I assume ) contain the certificate and private key for the instance, along with the CA certificate. That's why you use it both as a keystore and a truststore in your config. Unfortunately, Kibana doesn't currently support PKCS12 keystores so that you have to export the CA certificate to a pem encoded certificate and use that in your kibana configuration.

So in concrete steps:

1. Export the CA certificate from the PKCS12 keystore

   openssl pkcs12 -in client01.p12 -cacerts -nokeys -out elastic-ca.pem
   

2. Copy elastic-ca.pem to /etc/kibana/

3. Configure kibana to trust this certificate authority by configuring

   elasticsearch.ssl.certificateAuthorities: [ "/etc/kibana/elastic-ca.pem" ]
   

4. Restart Kibana

[mlemartien (Jacques Clement)]

Wow Ionnnis, thank you soooo much. You made my day!!! I have been struggling with this for two days and was getting desperate.

I did create the p12 keystores with certutil indeed, but did not know kibana could not use it as-is.

Thanks again!