Kibana 6.5 Status code 500


(Amit Hora) #1

After upgrading ES/Kibana from version 5.6 to 6.5 I am facing few problems. When i try to access Kibana URL I get this error
image

Below are the logs in Kibana
{"type":"log","@timestamp":"2018-12-06T05:55:28Z","tags":["listening","info"],"pid":31181,"message":"Server running at https://XXXX:5601"}
{"type":"log","@timestamp":"2018-12-06T05:55:30Z","tags":["status","plugin:security@6.5.1","info"],"pid":31181,"state":"green","message":"Status changed from yellow to green - Ready","prevState":"yellow","prevMsg":"Waiting for Elasticsearch"}
{"type":"log","@timestamp":"2018-12-06T05:55:30Z","tags":["status","plugin:spaces@6.5.1","info"],"pid":31181,"state":"green","message":"Status changed from yellow to green - Ready","prevState":"yellow","prevMsg":"Waiting for Elasticsearch"}

When I check the logs in Elasticsearch , I see something like this

[2018-12-06T05:55:59,281][WARN ][o.e.x.s.t.n.SecurityNetty4HttpServerTransport] [stage-es4] caught exception while handling client http traffic, closing connection [id: 0xac05acc8, L:0.0.0.0/0.0.0.0:9200 ! R:/XXXX:27740]
io.netty.handler.codec.DecoderException: javax.net.ssl.SSLHandshakeException: null cert chain
        at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:472) ~[netty-codec-4.1.30.Final.jar:4.1.30.Final]
        at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:278) ~[netty-codec-4.1.30.Final.jar:4.1.30.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [netty-transport-4.1.30.Final.jar:4.1.30.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [netty-transport-4.1.30.Final.jar:4.1.30.Final]
        at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) [netty-transport-4.1.30.Final.jar:4.1.30.Final]
        at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1434) [netty-transport-4.1.30.Final.jar:4.1.30.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [netty-transport-4.1.30.Final.jar:4.1.30.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [netty-transport-4.1.30.Final.jar:4.1.30.Final]
        at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:965) [netty-transport-4.1.30.Final.jar:4.1.30.Final]
        at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) [netty-transport-4.1.30.Final.jar:4.1.30.Final]
        at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:644) [netty-transport-4.1.30.Final.jar:4.1.30.Final]
        at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:544) [netty-transport-4.1.30.Final.jar:4.1.30.Final]
        at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:498) [netty-transport-4.1.30.Final.jar:4.1.30.Final]
        at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:458) [netty-transport-4.1.30.Final.jar:4.1.30.Final]
        at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:897) [netty-common-4.1.30.Final.jar:4.1.30.Final]
        at java.lang.Thread.run(Thread.java:748) [?:1.8.0_191]
Caused by: javax.net.ssl.SSLHandshakeException: null cert chain
        at sun.security.ssl.Handshaker.checkThrown(Handshaker.java:1521) ~[?:?]
        at sun.security.ssl.SSLEngineImpl.checkTaskThrown(SSLEngineImpl.java:528) ~[?:?]
        at sun.security.ssl.SSLEngineImpl.readNetRecord(SSLEngineImpl.java:802) ~[?:?]
        at sun.security.ssl.SSLEngineImpl.unwrap(SSLEngineImpl.java:766) ~[?:?]
        at javax.net.ssl.SSLEngine.unwrap(SSLEngine.java:624) ~[?:1.8.0_191]
        at io.netty.handler.ssl.SslHandler$SslEngineType$3.unwrap(SslHandler.java:294) ~[netty-handler-4.1.30.Final.jar:4.1.30.Final]
        at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1297) ~[netty-handler-4.1.30.Final.jar:4.1.30.Final]
        at io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1199) ~[netty-handler-4.1.30.Final.jar:4.1.30.Final]
        at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1243) ~[netty-handler-4.1.30.Final.jar:4.1.30.Final]
        at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:502) ~[netty-codec-4.1.30.Final.jar:4.1.30.Final]
        at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:441) ~[netty-codec-4.1.30.Final.jar:4.1.30.Final]
        ... 15 more

When I do a curl command with the same set of Self Signed certificates I can see the appropriate output

curl -k -XGET -u aa:aaa3 https://XXX:9200/_cluster/health --cert /usr/share/kibana/client-certs/client.crt --key /usr/share/kibana/client-certs/client.key

Any pointers will be of a great help!.

Regards,
Amit Singh Hora