Kibana 6.5.3 javax.net.ssl.SSLHandshakeException: null cert chain

Elastic Stack Kibana
1

Hello everyone.

I have an elastic cluster in 6.2.4, with the x-pack plugin, a licence, and full ssl required (http, transports)
A kibana is also installed and is authenticated with signed certificates

I am trying to install the latest version 6.5.3.
Everything works fine for Elastic, but no way to connect Kibana with ssl verification.

When I try to access kibana i get :
{"statusCode":500,"error":"Internal Server Error","message":"An internal server error occurred"}

Kibana logs :

{"type":"log","@timestamp":"2018-12-18T16:16:42Z","tags":["info","authentication"],"pid":11074,"message":"Authentication attempt failed: No Living connections"}
{"type":"error","@timestamp":"2018-12-18T16:16:41Z","tags":[],"pid":11074,"level":"error","error":{"message":"No Living connections","name":"Error","stack":"Error: No Living connections\n    at sendReqWithConnection (/usr/share/kibana/node_modules/elasticsearch/src/lib/transport.js:226:15)\n    at next (/usr/share/kibana/node_modules/elasticsearch/src/lib/connection_pool.js:214:7)\n    at _combinedTickCallback (internal/process/next_tick.js:132:7)\n    at process._tickDomainCallback (internal/process/next_tick.js:219:9)"},"url":{"protocol":null,"slashes":null,"auth":null,"host":null,"port":null,"hostname":null,"hash":null,"search":"","query":{},"pathname":"/app/infra","path":"/app/infra","href":"/app/infra"},"message":"No Living connections"}
{"type":"response","@timestamp":"2018-12-18T16:16:41Z","tags":[],"pid":11074,"method":"get","statusCode":500,"req":{"url":"/app/infra","method":"get","headers":{"host":"node-01","user-agent":"Mozilla/5.0 (Windows NT 6.1; rv:52.0) Gecko/20100101 20000523 Firefox/52.0","accept":"text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8","accept-language":"en-US,en;q=0.5","accept-encoding":"gzip, deflate, br","connection":"keep-alive","upgrade-insecure-requests":"1","cache-control":"max-age=0"},"remoteAddress":"10.10.10.1","userAgent":"10.10.10.5"},"res":{"statusCode":500,"responseTime":26,"contentLength":9},"message":"GET /app/infra 500 26ms - 9.0B"}

Elastic logs:

[2018-12-18T17:19:55,408][WARN ][o.e.x.s.t.n.SecurityNetty4HttpServerTransport] [node-01] caught exception while handling client http traffic, closing connection [id: 0x8426cb88, L:0.0.0.0/0.0.0.0:9200 ! R:/10.10.10.1:42482]
io.netty.handler.codec.DecoderException: javax.net.ssl.SSLHandshakeException: null cert chain
        at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:472) ~[netty-codec-4.1.30.Final.jar:4.1.30.Final]
.....
Caused by: javax.net.ssl.SSLHandshakeException: null cert chain
        at sun.security.ssl.Alerts.getSSLException(Alerts.java:192) ~[?:?]
        at sun.security.ssl.SSLEngineImpl.fatal(SSLEngineImpl.java:1666) ~[?:?]
        at sun.security.ssl.Handshaker.fatalSE(Handshaker.java:330) ~[?:?]
        at sun.security.ssl.Handshaker.fatalSE(Handshaker.java:318) ~[?:?]
        at sun.security.ssl.ServerHandshaker.clientCertificate(ServerHandshaker.java:1935) ~[?:?]
        at sun.security.ssl.ServerHandshaker.processMessage(ServerHandshaker.java:237) ~[?:?]
        at sun.security.ssl.Handshaker.processLoop(Handshaker.java:1052) ~[?:?]
        at sun.security.ssl.Handshaker$1.run(Handshaker.java:992) ~[?:?]
        at sun.security.ssl.Handshaker$1.run(Handshaker.java:989) ~[?:?]
        at java.security.AccessController.doPrivileged(Native Method) ~[?:1.8.0_181]
        at sun.security.ssl.Handshaker$DelegatedTask.run(Handshaker.java:1467) ~[?:?]
        at io.netty.handler.ssl.SslHandler.runDelegatedTasks(SslHandler.java:1457) ~[netty-handler-4.1.30.Final.jar:4.1.30.Final]
        at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1365) ~[netty-handler-4.1.30.Final.jar:4.1.30.Final]
        at io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1199) ~[netty-handler-4.1.30.Final.jar:4.1.30.Final]
        at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1243) ~[netty-handler-4.1.30.Final.jar:4.1.30.Final]
        at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:502) ~[netty-codec-4.1.30.Final.jar:4.1.30.Final]
        at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:441) ~[netty-codec-4.1.30.Final.jar:4.1.30.Final]
        ... 15 more

I can curl, using my certs :

curl --user kibana:password --cacert /etc/kibana/ssl/ca/ca.crt --cert /etc/kibana/ssl/kibana/kibana.crt --key /etc/kibana/ssl/kibana/kibana.key https://node-01:9200/_cat/nodes
10.10.10.100 3 64 3 0.92 0.70 0.54 mdi * node-01

In Kibana config :

server.ssl.enabled: true
server.ssl.certificate: /etc/kibana/ssl/kibana/kibana.crt
server.ssl.key: /etc/kibana/ssl/kibana/kibana.key

elasticsearch.ssl.certificate: /etc/kibana/ssl/kibana/kibana.crt
elasticsearch.ssl.key: /etc/kibana/ssl/kibana/kibana.key

elasticsearch.ssl.certificateAuthorities: ["/etc/kibana/ssl/ca/ca.crt" ]

elasticsearch.ssl.verificationMode: full

In Elastic config :

xpack.security.transport.ssl.enabled: true
xpack.security.transport.ssl.client_authentication: required
xpack.security.transport.ssl.verification_mode: certificate
xpack.security.transport.ssl.key: /etc/elasticsearch/ssl/elasticsearch/elastic.key
xpack.security.transport.ssl.certificate: /etc/elasticsearch/ssl/elasticsearch/elastic.crt
xpack.security.transport.ssl.certificate_authorities: ["/etc/elasticsearch/ssl/ca/ca.crt"]

xpack.security.http.ssl.enabled: true
xpack.security.http.ssl.client_authentication: required
xpack.security.http.ssl.key: /etc/elasticsearch/ssl/elasticsearch/elastic.key
xpack.security.http.ssl.certificate: /etc/elasticsearch/ssl/elasticsearch/elastic.crt
xpack.security.http.ssl.certificate_authorities: ["/etc/elasticsearch/ssl/ca/ca.crt"]

If I disable xpack.security.http.ssl.client_authentication and set it to optional or none, kibana is working correctly. But i don't want to disable it.

It's strange as I am using the same certificates from before and the curl is working with thoses certs..

Do you have any idea how to fix this ?

Thank you very much.

2

@Larry_Gregory/ @Brandon_Kobel - our security people can help here when they get some time.

Cheers
Rashmi

1 Like
3

Hey @totsugeki, 6.4 introduced a change in behavior when Kibana send it's certificates for certain ES connections. https://github.com/elastic/kibana/issues/22643 provides some context for what you're experiencing, and as of 6.5 you can use the elasticsearch.ssl.alwaysPresentCertificate: true setting in your kibana.yml to always present the certificates to ES; however, care should be used when setting this as if you're using the PKI realm in ES to authenticate users, in certain configurations this can lead to requests from your end-user being authenticated as the internal Kibana system user.

1 Like
4

Than you @Brandon_Kobel.

I missed this change in behavior (where could I have read about this ?)
So if I understand correctly :

If we set xpack.security.http.ssl.client_authentication: required in elasticsearch, it is impossible to use kibana without elasticsearch.ssl.alwaysPresentCertificate: true because elasticsearch is expecting a client certificat.

in certain configurations this can lead to requests from your end-user being authenticated as the internal Kibana system user.

Do you have more informations about this ?
What are the security implication ?

Thank you very much !

5

Do you have more informations about this ?
What are the security implication?

This is only an issue when using the PKI Realm in Elasticsearch, and the certificate which Kibana presents to Elasticsearch is used to authenticate the Kibana server's internal user. There are two types of requests which Kibana makes to Elasticsearch, those which it intends to make as Kibana server's internal user (by default the kibana user`), and those that it proxies to Elasticsearch on behalf of the end-user which we intend to make using the end-users credentials. On these proxied requests, if we're presenting the certificate and the end-user's certificates, we have to have the realms in Elasticsearch ordered in a specific way so that the end-users credentials take precedent over the certificate, or else we risk these proxied requests being executing on behalf of the Kibana server internal user.

6

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.