Hello everyone.
I have an elastic cluster in 6.2.4, with the x-pack plugin, a licence, and full ssl required (http, transports)
A kibana is also installed and is authenticated with signed certificates
I am trying to install the latest version 6.5.3.
Everything works fine for Elastic, but no way to connect Kibana with ssl verification.
When I try to access kibana i get :
{"statusCode":500,"error":"Internal Server Error","message":"An internal server error occurred"}
Kibana logs :
{"type":"log","@timestamp":"2018-12-18T16:16:42Z","tags":["info","authentication"],"pid":11074,"message":"Authentication attempt failed: No Living connections"}
{"type":"error","@timestamp":"2018-12-18T16:16:41Z","tags":[],"pid":11074,"level":"error","error":{"message":"No Living connections","name":"Error","stack":"Error: No Living connections\n at sendReqWithConnection (/usr/share/kibana/node_modules/elasticsearch/src/lib/transport.js:226:15)\n at next (/usr/share/kibana/node_modules/elasticsearch/src/lib/connection_pool.js:214:7)\n at _combinedTickCallback (internal/process/next_tick.js:132:7)\n at process._tickDomainCallback (internal/process/next_tick.js:219:9)"},"url":{"protocol":null,"slashes":null,"auth":null,"host":null,"port":null,"hostname":null,"hash":null,"search":"","query":{},"pathname":"/app/infra","path":"/app/infra","href":"/app/infra"},"message":"No Living connections"}
{"type":"response","@timestamp":"2018-12-18T16:16:41Z","tags":[],"pid":11074,"method":"get","statusCode":500,"req":{"url":"/app/infra","method":"get","headers":{"host":"node-01","user-agent":"Mozilla/5.0 (Windows NT 6.1; rv:52.0) Gecko/20100101 20000523 Firefox/52.0","accept":"text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8","accept-language":"en-US,en;q=0.5","accept-encoding":"gzip, deflate, br","connection":"keep-alive","upgrade-insecure-requests":"1","cache-control":"max-age=0"},"remoteAddress":"10.10.10.1","userAgent":"10.10.10.5"},"res":{"statusCode":500,"responseTime":26,"contentLength":9},"message":"GET /app/infra 500 26ms - 9.0B"}
Elastic logs:
[2018-12-18T17:19:55,408][WARN ][o.e.x.s.t.n.SecurityNetty4HttpServerTransport] [node-01] caught exception while handling client http traffic, closing connection [id: 0x8426cb88, L:0.0.0.0/0.0.0.0:9200 ! R:/10.10.10.1:42482]
io.netty.handler.codec.DecoderException: javax.net.ssl.SSLHandshakeException: null cert chain
at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:472) ~[netty-codec-4.1.30.Final.jar:4.1.30.Final]
.....
Caused by: javax.net.ssl.SSLHandshakeException: null cert chain
at sun.security.ssl.Alerts.getSSLException(Alerts.java:192) ~[?:?]
at sun.security.ssl.SSLEngineImpl.fatal(SSLEngineImpl.java:1666) ~[?:?]
at sun.security.ssl.Handshaker.fatalSE(Handshaker.java:330) ~[?:?]
at sun.security.ssl.Handshaker.fatalSE(Handshaker.java:318) ~[?:?]
at sun.security.ssl.ServerHandshaker.clientCertificate(ServerHandshaker.java:1935) ~[?:?]
at sun.security.ssl.ServerHandshaker.processMessage(ServerHandshaker.java:237) ~[?:?]
at sun.security.ssl.Handshaker.processLoop(Handshaker.java:1052) ~[?:?]
at sun.security.ssl.Handshaker$1.run(Handshaker.java:992) ~[?:?]
at sun.security.ssl.Handshaker$1.run(Handshaker.java:989) ~[?:?]
at java.security.AccessController.doPrivileged(Native Method) ~[?:1.8.0_181]
at sun.security.ssl.Handshaker$DelegatedTask.run(Handshaker.java:1467) ~[?:?]
at io.netty.handler.ssl.SslHandler.runDelegatedTasks(SslHandler.java:1457) ~[netty-handler-4.1.30.Final.jar:4.1.30.Final]
at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1365) ~[netty-handler-4.1.30.Final.jar:4.1.30.Final]
at io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1199) ~[netty-handler-4.1.30.Final.jar:4.1.30.Final]
at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1243) ~[netty-handler-4.1.30.Final.jar:4.1.30.Final]
at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:502) ~[netty-codec-4.1.30.Final.jar:4.1.30.Final]
at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:441) ~[netty-codec-4.1.30.Final.jar:4.1.30.Final]
... 15 more
I can curl, using my certs :
curl --user kibana:password --cacert /etc/kibana/ssl/ca/ca.crt --cert /etc/kibana/ssl/kibana/kibana.crt --key /etc/kibana/ssl/kibana/kibana.key https://node-01:9200/_cat/nodes
10.10.10.100 3 64 3 0.92 0.70 0.54 mdi * node-01
In Kibana config :
server.ssl.enabled: true
server.ssl.certificate: /etc/kibana/ssl/kibana/kibana.crt
server.ssl.key: /etc/kibana/ssl/kibana/kibana.key
elasticsearch.ssl.certificate: /etc/kibana/ssl/kibana/kibana.crt
elasticsearch.ssl.key: /etc/kibana/ssl/kibana/kibana.key
elasticsearch.ssl.certificateAuthorities: ["/etc/kibana/ssl/ca/ca.crt" ]
elasticsearch.ssl.verificationMode: full
In Elastic config :
xpack.security.transport.ssl.enabled: true
xpack.security.transport.ssl.client_authentication: required
xpack.security.transport.ssl.verification_mode: certificate
xpack.security.transport.ssl.key: /etc/elasticsearch/ssl/elasticsearch/elastic.key
xpack.security.transport.ssl.certificate: /etc/elasticsearch/ssl/elasticsearch/elastic.crt
xpack.security.transport.ssl.certificate_authorities: ["/etc/elasticsearch/ssl/ca/ca.crt"]
xpack.security.http.ssl.enabled: true
xpack.security.http.ssl.client_authentication: required
xpack.security.http.ssl.key: /etc/elasticsearch/ssl/elasticsearch/elastic.key
xpack.security.http.ssl.certificate: /etc/elasticsearch/ssl/elasticsearch/elastic.crt
xpack.security.http.ssl.certificate_authorities: ["/etc/elasticsearch/ssl/ca/ca.crt"]
If I disable xpack.security.http.ssl.client_authentication and set it to optional or none, kibana is working correctly. But i don't want to disable it.
It's strange as I am using the same certificates from before and the curl is working with thoses certs..
Do you have any idea how to fix this ?
Thank you very much.