Below is kibana.yml and elasticsearch.yml configuration
kibana.yml
server.host: "myip"
elasticsearch.url: "https://myip:9200"
elasticsearch.username: "kibana"
elasticsearch.password: "password"
elasticsearch.ssl.certificateAuthorities: [ "/etc/kibana/elastic-ca.pem" ]
elasticsearch.yml
cluster.name: elastic-cluster
node.name: node-1
path.data: /var/lib/elasticsearch
path.logs: /var/log/elasticsearch
network.host: myip
action.auto_create_index: .security,.monitoring*,.watches,.triggered_watches,.watcher-history*,.ml*
xpack.security.transport.ssl.enabled: true
xpack.security.transport.ssl.verification_mode: certificate
xpack.security.transport.ssl.keystore.path: config/certs/elastic-certificates.p12
xpack.security.transport.ssl.truststore.path: config/certs/elastic-certificates.p12
xpack.security.http.ssl.enabled: true
xpack.security.http.ssl.keystore.path: config/certs/elastic-certificates.p12
xpack.security.http.ssl.truststore.path: config/certs/elastic-certificates.p12
Below is the error which is shown in the elasticsearch log.
less /var/log/elasticsearch/elastic-cluster.log
[2018-04-24T18:26:30,326][WARN ][o.e.x.s.t.n.SecurityNetty4HttpServerTransport] [node-1] caught exception while handling client http traffic, closing connection [id: 0x897005a0, L:0.0.0.0/0.0.0.0:9200 ! R:/ip:58318]
io.netty.handler.codec.DecoderException: io.netty.handler.ssl.NotSslRecordException: not an SSL/TLS record: 474554202f20485454502f312e310d0a486f73743a2031302e3136392e33332e3139383a393230300d0a557365722d4167656e743a20476f2d687474702d636c69656e742f312e310d0a4163636570743a206170706c69636174696f6e2f6a736f6e0d0a4163636570742d456e636f64696e673a20677a69700d0a0d0a
at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:459) ~[netty-codec-4.1.16.Final.jar:4.1.16.Final]
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:265) ~[netty-codec-4.1.16.Final.jar:4.1.16.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1359) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:935) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:134) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:645) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:545) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:499) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:459) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) [netty-common-4.1.16.Final.jar:4.1.16.Final]
at java.lang.Thread.run(Thread.java:745) [?:1.8.0_91]
Caused by: io.netty.handler.ssl.NotSslRecordException: not an SSL/TLS record: 474554202f20485454502f312e310d0a486f73743a2031302e3136392e33332e3139383a393230300d0a557365722d4167656e743a20476f2d687474702d636c69656e742f312e310d0a4163636570743a206170706c69636174696f6e2f6a736f6e0d0a4163636570742d456e636f64696e673a20677a69700d0a0d0a
at io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1106) ~[?:?]
at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1162) ~[?:?]
at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:489) ~[?:?]
at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:428) ~[?:?]
... 15 more
Logs available in kibana
less /var/log/kibana/kibana.stdout
{"type":"log","@timestamp":"2018-04-24T13:06:14Z","tags":["warning","elasticsearch","admin"],"pid":9965,"message":"No living connections"}
{"type":"log","@timestamp":"2018-04-24T13:06:17Z","tags":["warning","elasticsearch","admin"],"pid":9965,"message":"Unable to revive connection: https://myip:9200/"}
{"type":"log","@timestamp":"2018-04-24T13:06:17Z","tags":["warning","elasticsearch","admin"],"pid":9965,"message":"No living connections"}
{"type":"log","@timestamp":"2018-04-24T13:06:19Z","tags":["warning","elasticsearch","admin"],"pid":9965,"message":"Unable to revive connection: https://myip:9200/"}
{"type":"log","@timestamp":"2018-04-24T13:06:19Z","tags":["warning","elasticsearch","admin"],"pid":9965,"message":"No living connections"}
{"type":"log","@timestamp":"2018-04-24T13:06:22Z","tags":["warning","elasticsearch","admin"],"pid":9965,"message":"Unable to revive connection: https://myip:9200/"}
{"type":"log","@timestamp":"2018-04-24T13:06:22Z","tags":["warning","elasticsearch","admin"],"pid":9965,"message":"No living connections"}
{"type":"log","@timestamp":"2018-04-24T13:06:24Z","tags":["warning","elasticsearch","admin"],"pid":9965,"message":"Unable to revive connection: https://myip:9200/"}
{"type":"log","@timestamp":"2018-04-24T13:06:24Z","tags":["warning","elasticsearch","admin"],"pid":9965,"message":"No living connections"}
{"type":"log","@timestamp":"2018-04-24T13:06:27Z","tags":["warning","elasticsearch","admin"],"pid":9965,"message":"Unable to revive connection: https://myip:9200/"}
{"type":"log","@timestamp":"2018-04-24T13:06:27Z","tags":["warning","elasticsearch","admin"],"pid":9965,"message":"No living connections"}
(END)