Certificates between Elasticsearch/Kibana RHEL VM and certificate authority Windows server

[rebellionne (mtz)]

Hi everybody
After days of research, I finally decided to ask for help. I must precise that working with certificates is quite new to me especially on Linux environment.

• I run Elasticsearch and Kibana 7.15.0 on the same RHEL7.9 Linux VM
  -My VM is part of a Windows domain and the DC is also the certificate authority server (Windows 2019)
  -On my VM, I have run: ./bin/elasticsearch-certutil http, hostnames are .machine.domain.com
  -I have transferred the .ZIPfile generated to the Windows CA server
  -I have run: certreq -submit -attrib "certificatetemplate:webserver", selected the .CSR file from the .ZIP one and transferred back to the Linux VM the .CER file generated by the CA server
  -I have renamed the .CER file to a .PEM one and copied it to my Elasticsearch and Kibana certs folder with the .KEY file (which was also in the .ZIP file)
  -On the certificate authority server, I also backed up the CA certificate ( private key and CA certificate) as a CAserver.p12 and transferred it to the Linux VM
  -There I've run : openssl pkcs12 -in CAserver.p12 -nokeys -out CAserver.pem
  -I've copied the CAserver.pem to the Elasticsearch, Kibana and pem /etc/pki/ca-trust/source/anchors/
  -Elasticsearch.yml and kibana.yml use those files
  -Elasticsearch starts correctly, I can access it from Edge on a Windows machine, I can successfully run curl -X GET "https://elastic.machine.domain.com:9200" -u elastic:password
  -But when I start Kibana it's stuck with the message "Unable to retrieve version information from Elasticsearch nodes. socket hang up"
  and my Elasticsearch immediately crashes with that message:

[2021-10-18T14:57:44,392][WARN ][o.e.c.s.DiagnosticTrustManager] [elastic] failed to establish trust with client at []; the client provided a certificate with subject name [CN=machine,DC=domain,DC=com] and fingerprint [6657021da561d7fae8e86045e938d5d4a4ba7da3]; the certificate is issued by [CN=serverCA,DC=domain,DC=com]; the certificate is signed by (subject [CN=serverCA,DC=domain,DC=com] fingerprint [1ca79d4f162a556234119039f40eea6c43b95989] {trusted issuer}) which is self-issued; the [CN=serverCA,DC=domain,DC=com] certificate is trusted in this ssl context ([xpack.security.http.ssl])
sun.security.validator.ValidatorException: Extended key usage does not permit use for TLS client authentication

-I understand that kibana cannot communicate with elasticsearch (on the same machine)
-I have googled it a bit and only found that this error happens because my certificate should be for both clientAuth and serverAuth which is not the case here, as the template webserver on Windows is a serverAuth only. No idea how to get a certificate for both...if that's the reason why it's crashing...
Many thanks for your support.

[TimV (Tim Vernum)]

It sounds like you have configured the TLS connection between Kibana and Elasticsearch to use mTLS ("mutual TLS", aka TLS Client Authentication, aka Client Certificates).

That's a valid & supported option, but it's not the most common setup people use - is it really what you want. There's no product level requirement for you to do that - you should only do it if you have a specific reason to want it.

@rebellionne

I have googled it a bit and only found that this error happens because my certificate should be for both clientAuth and serverAuth which is not the case here, as the template webserver on Windows is a serverAuth only. No idea how to get a certificate for both...if that's the reason why it's crashing...

From what you have posted, yes that seems to be the root cause. Kibana (acting as a client to Elasticsearch) is presenting a certificate that cannot be used from a client so the handshake fails.

Per my comments above, I think the answer is to stop using a certificate on the client side, but if you really want one, you can use elasticsearch-certutil to generate one.