After upgrading to 7x certs dont work

Hello,

I had a functioning pki realm, and native realm with working certs. I upgraded to 7x from 6x and now suddenly I'm trying to test authentication using the xpac authenticate endpoint and those very same certs that were working before no longer get through. Do I need to recreate these certs from scratch due to some change in 7x? Heres a curl attempt:

curl https://localhost:9200/_xpack/security/_authenticate?pretty --key client.key --cert client.cer --cacert client-ca.cer -k -vvvv
*   Trying 127.0.0.1...
* TCP_NODELAY set
* Connected to localhost (127.0.0.1) port 9200 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
*   CAfile: client-ca.cer
  CApath: /etc/ssl/certs
* TLSv1.2 (OUT), TLS handshake, Client hello (1):
* TLSv1.2 (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS handshake, Certificate (11):
* TLSv1.2 (IN), TLS handshake, Server key exchange (12):
* TLSv1.2 (IN), TLS handshake, Request CERT (13):
* TLSv1.2 (IN), TLS handshake, Server finished (14):
* TLSv1.2 (OUT), TLS handshake, Certificate (11):
* TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
* TLSv1.2 (OUT), TLS handshake, CERT verify (15):
* TLSv1.2 (OUT), TLS change cipher, Client hello (1):
* TLSv1.2 (OUT), TLS handshake, Finished (20):
* TLSv1.2 (IN), TLS alert, Server hello (2):
* error:14094416:SSL routines:ssl3_read_bytes:sslv3 alert certificate unknown
* stopped the pause stream!
* Closing connection 0
curl: (35) error:14094416:SSL routines:ssl3_read_bytes:sslv3 alert certificate unknown

I can actually get by with just the client-ca.cer and passing creds via basic auth. So I feel like the ca is still in order. the cert and key were generated from the same p12 file so it makes no sense that suddenly the client.cer is getting rejected. Thank you for your help.

Including this if interested, just the ca, and of course it complains about credentials, which i can get passed if i just provide them:

curl https://localhost:9200/_xpack/security/_authenticate?pretty --cacert client-ca.cer -k -vvvv
*   Trying 127.0.0.1...
* TCP_NODELAY set
* Connected to localhost (127.0.0.1) port 9200 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
*   CAfile: client-ca.cer
  CApath: /etc/ssl/certs
* TLSv1.2 (OUT), TLS handshake, Client hello (1):
* TLSv1.2 (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS handshake, Certificate (11):
* TLSv1.2 (IN), TLS handshake, Server key exchange (12):
* TLSv1.2 (IN), TLS handshake, Request CERT (13):
* TLSv1.2 (IN), TLS handshake, Server finished (14):
* TLSv1.2 (OUT), TLS handshake, Certificate (11):
* TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
* TLSv1.2 (OUT), TLS change cipher, Client hello (1):
* TLSv1.2 (OUT), TLS handshake, Finished (20):
* TLSv1.2 (IN), TLS handshake, Finished (20):
* SSL connection using TLSv1.2 / ECDHE-RSA-AES256-SHA384
* ALPN, server did not agree to a protocol
* Server certificate:
*  subject: CN=instance
*  start date: May 14 14:55:23 2019 GMT
*  expire date: May 13 14:55:23 2022 GMT
*  issuer: CN=Elastic Certificate Tool Autogenerated CA
*  SSL certificate verify result: self signed certificate in certificate chain (19), continuing anyway.
> GET /_xpack/security/_authenticate?pretty HTTP/1.1
> Host: localhost:9200
> User-Agent: curl/7.58.0
> Accept: */*
>
< HTTP/1.1 401 Unauthorized
< WWW-Authenticate: Bearer realm="security"
< WWW-Authenticate: ApiKey
< WWW-Authenticate: Basic realm="security" charset="UTF-8"
< content-type: application/json; charset=UTF-8
< content-length: 746
<
{
  "error" : {
    "root_cause" : [
      {
        "type" : "security_exception",
        "reason" : "missing authentication credentials for REST request [/_xpack/security/_authenticate?pretty]",
        "header" : {
          "WWW-Authenticate" : [
            "Bearer realm=\"security\"",
            "ApiKey",
            "Basic realm=\"security\" charset=\"UTF-8\""
          ]
        }
      }
    ],
    "type" : "security_exception",
    "reason" : "missing authentication credentials for REST request [/_xpack/security/_authenticate?pretty]",
    "header" : {
      "WWW-Authenticate" : [
        "Bearer realm=\"security\"",
        "ApiKey",
        "Basic realm=\"security\" charset=\"UTF-8\""
      ]
    }
  },
  "status" : 401
}
* Connection #0 to host localhost left intact

Redid cert steps just as a sanity check. Same results. Heres openssl attempt for the sake of thoroughness note the exact same error:

openssl s_client -connect localhost:9200 -key client.key -cert client.cer -CAfile client-ca.cer
CONNECTED(00000003)
depth=1 CN = Elastic Certificate Tool Autogenerated CA
verify error:num=19:self signed certificate in certificate chain
**140560582279616:error:14094416:SSL routines:ssl3_read_bytes:sslv3 alert certificate unknown:../ssl/record/rec_layer_s3.c:1399:SSL alert number 46**
---
Certificate chain
 0 s:/CN=instance
   i:/CN=Elastic Certificate Tool Autogenerated CA
 1 s:/CN=Elastic Certificate Tool Autogenerated CA
   i:/CN=Elastic Certificate Tool Autogenerated CA
---
Server certificate
-----BEGIN CERTIFICATE-----
***removed***
-----END CERTIFICATE-----
subject=/CN=instance
issuer=/CN=Elastic Certificate Tool Autogenerated CA
---
Acceptable client certificate CA names
/CN=instance
/CN=Elastic Certificate Tool Autogenerated CA
Client Certificate Types: ECDSA sign, RSA sign, DSA sign
Requested Signature Algorithms: ECDSA+SHA256:ECDSA+SHA384:ECDSA+SHA512:0x04+0x08:0x05+0x08:0x06+0x08:0x09+0x08:0x0A+0x08:0x0B+0x08:RSA+SHA256:RSA+SHA384:RSA+SHA512:DSA+SHA256:ECDSA+SHA224:RSA+SHA224:DSA+SHA224:ECDSA+SHA1:RSA+SHA1:DSA+SHA1
Shared Requested Signature Algorithms: ECDSA+SHA256:ECDSA+SHA384:ECDSA+SHA512:RSA+SHA256:RSA+SHA384:RSA+SHA512:DSA+SHA256:ECDSA+SHA224:RSA+SHA224:DSA+SHA224:ECDSA+SHA1:RSA+SHA1:DSA+SHA1
Peer signing digest: SHA512
Server Temp Key: ECDH, P-256, 256 bits
---
SSL handshake has read 2228 bytes and written 2368 bytes
Verification error: self signed certificate in certificate chain
---
New, TLSv1.2, Cipher is ECDHE-RSA-AES256-SHA384
Server public key is 2048 bit
Secure Renegotiation IS supported
Compression: NONE
Expansion: NONE
No ALPN negotiated
SSL-Session:
    Protocol  : TLSv1.2
    Cipher    : ECDHE-RSA-AES256-SHA384
    Session-ID: 83F14F7B0724A68418F5205BCCB194A2C1286B86C73837B28BF4E8160EBF4514
    Session-ID-ctx:
    Master-Key: 0C354D32C3665C35E64C5D646983AAAC1F6AE36F4181B325ACC24AE863F5153CAAAF5DE71A3931814C75AE1BFF7F647C
    PSK identity: None
    PSK identity hint: None
    SRP username: None
    Start Time: 1559322482
    Timeout   : 7200 (sec)
    Verify return code: 19 (self signed certificate in certificate chain)
    Extended master secret: yes
---

these are basic steps that can be found here:
https://www.elastic.co/guide/en/elasticsearch/reference/current/configuring-tls.html

I'm using the basic ca thats generated by the cert util, to create the cert I'm trying to use to connect back into elastic.

I'd also like to add that, i got a complaint from curl saying openssl cannot support tls 1.3, so I thought maybe its all working and just try the certs with kibana, but kibana is getting it as well:

Request error, retrying
HEAD https://localhost:9200/ => write EPROTO 140523106330432:error:14094416:SSL routines:ssl3_read_bytes:sslv3 alert certificate unknown:../deps/openssl/openssl/ssl/record/rec_layer_s3.c:1407:SSL alert number 46

and the dreaded no living connections:

[20:05:47.953] [warning][admin][elasticsearch] Unable to revive connection: https://localhost:9200/
  log   [20:05:47.954] [warning][admin][elasticsearch] No living connections
  log   [20:05:47.958] [warning][admin][elasticsearch] Unable to revive connection: https://localhost:9200/
  log   [20:05:47.958] [warning][admin][elasticsearch] No living connections
  log   [20:05:47.959] [warning][task_manager] PollError No Living connections

and if it helps, kibana config:

server.host: "598717Ubuntu1"

elasticsearch.hosts: ["https://localhost:9200"]

xpack.security.enabled: true
server.ssl.enabled: true
server.ssl.key: /etc/kibana/certs/server.key
server.ssl.certificate: /etc/kibana/certs/server.crt
elasticsearch.ssl.verificationMode: certificate
elasticsearch.ssl.certificateAuthorities: /etc/kibana/certs/client-ca.cer
elasticsearch.ssl.certificate: /etc/kibana/certs/client.cer
elasticsearch.ssl.key: /etc/kibana/certs/client.key

elastic conf (yes I updated the realm portion to match 7x):

xpack.security.enabled: true
xpack.security.transport.ssl.verification_mode: certificate
xpack.security.transport.ssl.keystore.path: /etc/elasticsearch/elastic-certificates.p12
xpack.security.transport.ssl.truststore.path: /etc/elasticsearch/elastic-certificates.p12
xpack.security.http.ssl.enabled: true
xpack.security.http.ssl.keystore.path: /etc/elasticsearch/elastic-certificates.p12
xpack.security.http.ssl.truststore.path: /etc/elasticsearch/elastic-certificates.p12
xpack.security.http.ssl.client_authentication: optional
xpack.security.authc.realms.pki.pki1.order: 1
xpack.security.authc.realms.native.native1.order: 2

issue seems to be resolved...I actually dont know why I just kept tinkering with it until the problem went away. The best I can say for others who run into this:

curl/openssl, might not support tls 1.3 based on the errors above, perhaps the certs worked all along, but curl/openssl couldn't show it or use them they only got in with basic auth.

Kibana is using those client certs, same ones that failed with curl/openssl, and its connecting with no basic auth. My guess, is that the latest 7.1 update is using TLS 1.3, curl/openssl cannot so they were being misleading. I tried to force curl to use the TLS 1.3 protocol and it said "openssl was not built to use TLS1.3" or something like that.

One more thing. The responses of curl/openssl led me to believe that I had to use basic auth with Kibana. This was failing because I was trying to replicate what I did with curl, ie, ca-cert and basic auth to get in. But according to documentation:

These files [elasticsearch.ssl.certificate: and elasticsearch.ssl.key] are used to verify 
the identity of Kibana to Elasticsearch and are required when xpack.ssl.verification_mode 
in Elasticsearch is set to either certificate or full

I think kibana was failing because I removed the cert/key lines from the config and tried to replace the with basic auth. In the end I should have just let kibana use the certs instead of basic auth with the ca-cert. It was probably working but curl/openssl made me think kibana wouldn't work.

TLDR: if TLS 1.3 w/ certs, just because curl/openssl doesnt work, doesn't mean kibana doesn't work.

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.