Connecting Kibana to Elasticsearch over HTTPS: unsupported certificate purpose

Elastic Stack Kibana
1

Hey folks,

I'm in the process of setting up my first larger ES cluster, and first time going full HTTPS across the board. Bit of background: 40ish ES nodes, with 2 Kibana nodes. Using an internal root CA / Intermediate CA / signed cert setup. Originally was doing this manually with openssl, now managing with saltstack.

Certs have all been generated with:

  • keyUsage: "critical, digitalSignature, nonRepudiation, keyEncipherment, keyAgreement"
  • extendedKeyUsage: "serverAuth, clientAuth"

All the ES nodes are working as expected. No errors in the logs, curl commands pulling back cluster health show all nodes online. Great.

So now I'm attempting the next step, which is to get Kibana connected to ES. I chose to install an ES "Coordinating Only" node directly on the two Kibana servers; these are communicating to the rest of the ES cluster fine over TLS (no log errors, showing up in curl _cluster/health as expected, etc).

I have a very simple kibana.yml at the moment. It looks like this:

# kibana Config file

elasticsearch.hosts: 'https://WEB1:9200'

elasticsearch.username: 'kibana'
elasticsearch.password: 'sanitizedpassword'

path.data:  /opt/kibana
logging.dest: /opt/kibana/kibana.log

server.host: WEB1
server.name: WEB1

elasticsearch.ssl.certificateAuthorities: [ "/etc/kibana/Root_Chain.cert" ]
elasticsearch.ssl.verificationMode: certificate

xpack.security.encryptionKey: “sanitizedlongstring”

Note I'm not doing TLS browser -> Kibana yet. One step at a time.

My understanding of the above config is that it should connect to ES over HTTPS, and because I reference the Root Chain it should be trusted and all should be well. I set ssl.verificationMode to certificate simply to eliminate the possibility of hostname verification (although I have the SAN's in the certs set to include fqdn and short hostnames, but I digress).

The Problem:

Kibana can't connect to ES. Here's what I believe is the relevant error in kibana.log:

{"type":"log","@timestamp":"2019-06-21T14:10:46Z","tags":["error","elasticsearch","admin"],"pid":27684,"message":"Request error, retrying\nHEAD https://web1:9200/ => unsupported certificate purpose"}

Seems pretty cut and dry. The blog here: https://www.elastic.co/blog/elasticsearch-security-configure-tls-ssl-pki-authentication indicates that certs used for ES in general need both clientAuth and serverAuth indicated in the x509 Extended Key Usage area. Which I have though; this was needed to get ES itself talking.

If I do an 'openssl x509 -in web1.cert -noout -text -purpose' here is the output (trimmed a bit for readability) :

Certificate:
    Data:
        Version: 3 (0x2)
        Serial Number: 543860926803148573 (0x78c2ea1a550ff1d)
    Signature Algorithm: sha256WithRSAEncryption
        Issuer: C=CA, CN=saltstack Intermediate CA, L=Ottawa, ST=Ontario
        Validity
            Not Before: Jun 21 14:08:12 2019 GMT
            Not After : Jun 20 14:08:12 2021 GMT
        Subject: C=CA, CN=web1.local, L=Ottawa, ST=Ontario, O=xxx, OU=xxx
        Subject Public Key Info:
            Public Key Algorithm: rsaEncryption
                Public-Key: (2048 bit)
                Modulus:
 (removed)
        X509v3 extensions:
            X509v3 Basic Constraints: critical
                CA:FALSE
            X509v3 Key Usage:
                Digital Signature, Non Repudiation, Key Encipherment, Key Agreement
            X509v3 Extended Key Usage:
                TLS Web Server Authentication, TLS Web Client Authentication
            X509v3 Subject Key Identifier:
                CC:79:F3:21:45:28:71:7F:D2:23:CF:6A:A4:E8:82:35:07:F4:84:AA
            X509v3 Authority Key Identifier:
                keyid:4E:60:5A:20:DD:79:D5:CA:9F:20:A9:4E:94:BD:3E:20:46:25:48:A5
                DirName:/C=CA/CN=saltstack Root CA/L=Ottawa/ST=Ontario/O=xxx/OU=Saltstack Certificate Authority
                serial:D2:46:AB:E0:1C:12:1C:6B

            X509v3 Subject Alternative Name:
                DNS:web1.local, DNS:web1, IP Address:172.16.1.1, IP Address:192.168.1.1
    Signature Algorithm: sha256WithRSAEncryption
 (removed)
Certificate purposes:
SSL client : Yes
SSL client CA : No
SSL server : Yes
SSL server CA : No
Netscape SSL server : Yes
Netscape SSL server CA : No
S/MIME signing : No
S/MIME signing CA : No
S/MIME encryption : No
S/MIME encryption CA : No
CRL signing : No
CRL signing CA : No
Any Purpose : Yes
Any Purpose CA : Yes
OCSP helper : Yes
OCSP helper CA : No
Time Stamp signing : No
Time Stamp signing CA : No

(Hopefully I formatted the above correctly).

So I THINK I've done everything as per documented requirements but I can't get Kibana to connect. Note that if I set elasticsearch.ssl.verificationMode to 'none' it DOES work; but this sort of bypasses the value of SSL.

Also, you'll probably realize this is all in-the-same-box communication so setting the above to none probably isn't the end of the world, but I'd really like to know why this isn't working.

Any help would be appreciated!

2

Hey @Chris_Murray, can you check the Elasticsearch logs of the coordinating node that you're connecting Kibana to? A lot of times, they contain the real error which isn't being surfaced to Kibana.

3

Thanks for the reply. Rechecked the ES logs, no errors whatsoever.

Now I'm wondering if there is a way to turn up the ES Log verbosity to log connection attempts...

4

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.