I have a three node cluster with Elastic 8.4.2. When all three nodes are up, it all appears fine. I built the cluster with a single node (audit1), but later added the other two nodes (audit2 & audit3). If I shutdown either audit2 or audit3, the cluster keeps on running fine with no observable issues. However, if I shutdown audit1, the entire cluster is no longer accessible. From other posts, I've looked at voting exclusions and none are there. I've made these three nodes as identical as I know how, but there appears to be some other thing that makes audit1 unique.
Anyone have any ideas?
Please share the elasticsearch.yml of each node.
cluster.name: UnclassTest
node.name: audit1
path.data: /var/lib/elasticsearch
path.logs: /var/log/elasticsearch
network.host: 0.0.0.0
http.port: 9200
discovery.seed_hosts:
- "audit-dev.u.l3"
- "auditsvr2.u.l3"
- "auditsvr3.u.l3"
cluster.initial_master_nodes:
- "audit-dev.u.l3"
- "auditsvr2.u.l3"
- "auditsvr3.u.l3"
node.roles: ["master","data","ingest"]
xpack.security.enabled: true
xpack.security.enrollment.enabled: true
xpack.security.http.ssl:
enabled: true
keystore.path: certs/http.p12
xpack.security.transport.ssl:
enabled: true
verification_mode: certificate
keystore.path: certs/transport.p12
truststore.path: certs/transport.p12
http.host: audit-dev.u.l3
transport.host: audit-dev.u.l3
The other nodes have identical yml files with the exception of the node.name, http.host, and transport.host. Those fields are specific to the node's hostname.
Is audit-dev your audit1 node?
What do you have in the logs fo the other nodes when you shutdown this node?
Also, share your configuration and logs using the preformatted option, the </> button in the editor, it makes easier to read the configuration.
elasticsearch received SSLhandshake exception: bad_certificate.
Surviving nodes are showing high disk watermark errors, but system df -h commands show no such problem.
Yes, audit-dev is audit1.
Where? From Which node? if you have this error you wouldn't have formed a cluster.
How much used and free space do you have in the elasticsearch data path?
You need to share the logs, without seeing the logs it is not possible to know what may be the issue.
This is the elasticsearch log contents from audit2 when audit1 was powered off.
````Preformatted text[2023-01-06T15:26:03,943][WARN ][o.e.x.s.t.n.SecurityNetty4HttpServerTransport] [audit2] received plaintext http traffic on an https channel, closing connection Netty4HttpChannel{localAddress=/10.16.106.196:9200, remoteAddress=/10.16.234.71:60744} [2023-01-06T15:26:08,366][WARN ][o.e.x.s.t.n.SecurityNetty4HttpServerTransport] [audit2] received plaintext http traffic on an https channel, closing connection Netty4HttpChannel{localAddress=/10.16.106.196:9200, remoteAddress=/10.16.24.98:52260} [2023-01-06T15:26:25,197][WARN ][o.e.x.s.t.n.SecurityNetty4HttpServerTransport] [audit2] received plaintext http traffic on an https channel, closing connection Netty4HttpChannel{localAddress=/10.16.106.196:9200, remoteAddress=/10.16.234.71:60862} [2023-01-06T15:26:34,351][WARN ][o.e.h.AbstractHttpServerTransport] [audit2] caught exception while handling client http traffic, closing connection Netty4HttpChannel{localAddress=/10.16.106.196:9200, remoteAddress=/10.16.234.71:60868} io.netty.handler.codec.DecoderException: javax.net.ssl.SSLHandshakeException: Received fatal alert: bad_certificate at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:480) ~[?:?] at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:279) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?] at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?] at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) ~[?:?] at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) ~[?:?] at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:722) ~[?:?] at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:623) ~[?:?] at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:586) ~[?:?] at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:496) ~[?:?] at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:995) ~[?:?] at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[?:?] at java.lang.Thread.run(Thread.java:833) ~[?:?] Caused by: javax.net.ssl.SSLHandshakeException: Received fatal alert: bad_certificate at sun.security.ssl.Alert.createSSLException(Alert.java:131) ~[?:?] at sun.security.ssl.Alert.createSSLException(Alert.java:117) ~[?:?] at sun.security.ssl.TransportContext.fatal(TransportContext.java:358) ~[?:?] at sun.security.ssl.Alert$AlertConsumer.consume(Alert.java:293) ~[?:?] at sun.security.ssl.TransportContext.dispatch(TransportContext.java:204) ~[?:?] at sun.security.ssl.SSLTransport.decode(SSLTransport.java:172) ~[?:?] at sun.security.ssl.SSLEngineImpl.decode(SSLEngineImpl.java:736) ~[?:?] at sun.security.ssl.SSLEngineImpl.readRecord(SSLEngineImpl.java:691) ~[?:?] at sun.security.ssl.SSLEngineImpl.unwrap(SSLEngineImpl.java:506) ~[?:?] at sun.security.ssl.SSLEngineImpl.unwrap(SSLEngineImpl.java:482) ~[?:?] at javax.net.ssl.SSLEngine.unwrap(SSLEngine.java:679) ~[?:?] at io.netty.handler.ssl.SslHandler$SslEngineType$3.unwrap(SslHandler.java:295) ~[?:?] at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1342) ~[?:?] at io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1235) ~[?:?] at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1284) ~[?:?] at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:510) ~[?:?] at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:449) ~[?:?] ... 16 more [2023-01-06T15:26:47,713][WARN ][o.e.x.s.t.n.SecurityNetty4HttpServerTransport] [audit2] received plaintext http traffic on an https channel, closing connection Netty4HttpChannel{localAddress=/10.16.106.196:9200, remoteAddress=/10.16.24.98:52858} [2023-01-06T15:26:59,066][WARN ][o.e.x.s.t.n.SecurityNetty4HttpServerTransport] [audit2] received plaintext http traffic on an https channel, closing connection Netty4HttpChannel{localAddress=/10.16.106.196:9200, remoteAddress=/10.16.234.71:60948} [2023-01-06T15:27:15,191][WARN ][o.e.x.s.t.n.SecurityNetty4HttpServerTransport] [audit2] received plaintext http traffic on an https channel, closing connection Netty4HttpChannel{localAddress=/10.16.106.196:9200, remoteAddress=/10.16.234.71:32962} [2023-01-06T15:27:19,307][WARN ][o.e.h.AbstractHttpServerTransport] [audit2] caught exception while handling client http traffic, closing connection Netty4HttpChannel{localAddress=/10.16.106.196:9200, remoteAddress=/10.16.234.71:32968} io.netty.handler.codec.DecoderException: javax.net.ssl.SSLHandshakeException: Received fatal alert: bad_certificate at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:480) ~[?:?] at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:279) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?] at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?] at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) ~[?:?] at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) ~[?:?] at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:722) ~[?:?] at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:623) ~[?:?] at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:586) ~[?:?] at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:496) ~[?:?] at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:995) ~[?:?] at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[?:?] at java.lang.Thread.run(Thread.java:833) ~[?:?] Caused by: javax.net.ssl.SSLHandshakeException: Received fatal alert: bad_certificate at sun.security.ssl.Alert.createSSLException(Alert.java:131) ~[?:?] at sun.security.ssl.Alert.createSSLException(Alert.java:117) ~[?:?] at sun.security.ssl.TransportContext.fatal(TransportContext.java:358) ~[?:?] at sun.security.ssl.Alert$AlertConsumer.consume(Alert.java:293) ~[?:?] at sun.security.ssl.TransportContext.dispatch(TransportContext.java:204) ~[?:?] at sun.security.ssl.SSLTransport.decode(SSLTransport.java:172) ~[?:?] at sun.security.ssl.SSLEngineImpl.decode(SSLEngineImpl.java:736) ~[?:?] at sun.security.ssl.SSLEngineImpl.readRecord(SSLEngineImpl.java:691) ~[?:?] at sun.security.ssl.SSLEngineImpl.unwrap(SSLEngineImpl.java:506) ~[?:?] at sun.security.ssl.SSLEngineImpl.unwrap(SSLEngineImpl.java:482) ~[?:?] at javax.net.ssl.SSLEngine.unwrap(SSLEngine.java:679) ~[?:?] at io.netty.handler.ssl.SslHandler$SslEngineType$3.unwrap(SslHandler.java:295) ~[?:?] at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1342) ~[?:?] at io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1235) ~[?:?] at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1284) ~[?:?] at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:510) ~[?:?] at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:449) ~[?:?] ... 16 more [2023-01-06T15:27:34,387][WARN ][o.e.x.s.t.n.SecurityNetty4HttpServerTransport] [audit2] received plaintext http traffic on an https channel, closing connection Netty4HttpChannel{localAddress=/10.16.106.196:9200, remoteAddress=/10.16.24.98:53608} [2023-01-06T15:27:44,875][WARN ][o.e.x.s.t.n.SecurityNetty4HttpServerTransport] [audit2] received plaintext http traffic on an https channel, closing connection Netty4HttpChannel{localAddress=/10.16.106.196:9200, remoteAddress=/10.16.234.71:33066} [2023-01-06T15:27:52,937][WARN ][o.e.h.AbstractHttpServerTransport] [audit2] caught exception while handling client http traffic, closing connection Netty4HttpChannel{localAddress=/10.16.106.196:9200, remoteAddress=/10.16.234.71:33072} io.netty.handler.codec.DecoderException: javax.net.ssl.SSLHandshakeException: Received fatal alert: bad_certificate at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:480) ~[?:?] at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:279) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?] at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?] at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) ~[?:?] at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) ~[?:?] at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:722) ~[?:?] at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:623) ~[?:?] at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:586) ~[?:?] at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:496) ~[?:?] at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:995) ~[?:?] at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[?:?] at java.lang.Thread.run(Thread.java:833) ~[?:?] Caused by: javax.net.ssl.SSLHandshakeException: Received fatal alert: bad_certificate at sun.security.ssl.Alert.createSSLException(Alert.java:131) ~[?:?] at sun.security.ssl.Alert.createSSLException(Alert.java:117) ~[?:?] at sun.security.ssl.TransportContext.fatal(TransportContext.java:358) ~[?:?] at sun.security.ssl.Alert$AlertConsumer.consume(Alert.java:293) ~[?:?] at sun.security.ssl.TransportContext.dispatch(TransportContext.java:204) ~[?:?] at sun.security.ssl.SSLTransport.decode(SSLTransport.java:172) ~[?:?] at sun.security.ssl.SSLEngineImpl.decode(SSLEngineImpl.java:736) ~[?:?] at sun.security.ssl.SSLEngineImpl.readRecord(SSLEngineImpl.java:691) ~[?:?] at sun.security.ssl.SSLEngineImpl.unwrap(SSLEngineImpl.java:506) ~[?:?] at sun.security.ssl.SSLEngineImpl.unwrap(SSLEngineImpl.java:482) ~[?:?] at javax.net.ssl.SSLEngine.unwrap(SSLEngine.java:679) ~[?:?] at io.netty.handler.ssl.SslHandler$SslEngineType$3.unwrap(SslHandler.java:295) ~[?:?] at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1342) ~[?:?] at io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1235) ~[?:?] at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1284) ~[?:?] at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:510) ~[?:?] at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:449) ~[?:?] ... 16 more [2023-01-06T15:28:02,792][WARN ][o.e.x.s.t.n.SecurityNetty4HttpServerTransport] [audit2] received plaintext http traffic on an https channel, closing connection Netty4HttpChannel{localAddress=/10.16.106.196:9200, remoteAddress=/10.16.234.71:33092} [2023-01-06T15:28:12,650][WARN ][o.e.x.s.t.n.SecurityNetty4HttpServerTransport] [audit2] received plaintext http traffic on an https channel, closing connection Netty4HttpChannel{localAddress=/10.16.106.196:9200, remoteAddress=/10.16.24.98:54194} [2023-01-06T15:28:26,435][WARN ][o.e.x.s.t.n.SecurityNetty4HttpServerTransport] [audit2] received plaintext http traffic on an https channel, closing connection Netty4HttpChannel{localAddress=/10.16.106.196:9200, remoteAddress=/10.16.234.71:33214} [2023-01-06T15:28:33,638][WARN ][o.e.x.s.t.n.SecurityNetty4HttpServerTransport] [audit2] received plaintext http traffic on an https channel, closing connection Netty4HttpChannel{localAddress=/10.16.106.196:9200, remoteAddress=/10.16.234.71:33222} [2023-01-06T15:28:51,169][WARN ][o.e.h.AbstractHttpServerTransport] [audit2] caught exception while handling client http traffic, closing connection Netty4HttpChannel{localAddress=/10.16.106.196:9200, remoteAddress=/10.16.234.71:33290} io.netty.handler.codec.DecoderException: javax.net.ssl.SSLHandshakeException: Received fatal alert: bad_certificate at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:480) ~[?:?] at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:279) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?] at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?] at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) ~[?:?] at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) ~[?:?] at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:722) ~[?:?] at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:623) ~[?:?] at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:586) ~[?:?] at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:496) ~[?:?] at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:995) ~[?:?] at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[?:?] at java.lang.Thread.run(Thread.java:833) ~[?:?] Caused by: javax.net.ssl.SSLHandshakeException: Received fatal alert: bad_certificate at sun.security.ssl.Alert.createSSLException(Alert.java:131) ~[?:?] at sun.security.ssl.Alert.createSSLException(Alert.java:117) ~[?:?] at sun.security.ssl.TransportContext.fatal(TransportContext.java:358) ~[?:?] at sun.security.ssl.Alert$AlertConsumer.consume(Alert.java:293) ~[?:?] at sun.security.ssl.TransportContext.dispatch(TransportContext.java:204) ~[?:?] at sun.security.ssl.SSLTransport.decode(SSLTransport.java:172) ~[?:?] at sun.security.ssl.SSLEngineImpl.decode(SSLEngineImpl.java:736) ~[?:?] at sun.security.ssl.SSLEngineImpl.readRecord(SSLEngineImpl.java:691) ~[?:?] at sun.security.ssl.SSLEngineImpl.unwrap(SSLEngineImpl.java:506) ~[?:?] at sun.security.ssl.SSLEngineImpl.unwrap(SSLEngineImpl.java:482) ~[?:?] at javax.net.ssl.SSLEngine.unwrap(SSLEngine.java:679) ~[?:?] at io.netty.handler.ssl.SslHandler$SslEngineType$3.unwrap(SslHandler.java:295) ~[?:?] at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1342) ~[?:?] at io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1235) ~[?:?] at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1284) ~[?:?] at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:510) ~[?:?] at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:449) ~[?:?] ... 16 more [2023-01-06T15:29:01,469][WARN ][o.e.x.s.t.n.SecurityNetty4HttpServerTransport] [audit2] received plaintext http traffic on an https channel, closing connection Netty4HttpChannel{localAddress=/10.16.106.196:9200, remoteAddress=/10.16.24.98:54966} [2023-01-06T15:29:09,661][WARN ][o.e.x.s.t.n.SecurityNetty4HttpServerTransport] [audit2] received plaintext http traffic on an https channel, closing connection Netty4HttpChannel{localAddress=/10.16.106.196:9200, remoteAddress=/10.16.234.71:33356} [2023-01-06T15:29:19,479][WARN ][o.e.x.s.t.n.SecurityNetty4HttpServerTransport] [audit2] received plaintext http traffic on an https channel, closing connection Netty4HttpChannel{localAddress=/10.16.106.196:9200, remoteAddress=/10.16.234.71:33416} [2023-01-06T15:29:21,832][WARN ][o.e.h.AbstractHttpServerTransport] [audit2] caught exception while handling client http traffic, closing connection Netty4HttpChannel{localAddress=/10.16.106.196:9200, remoteAddress=/10.16.234.71:33418} io.netty.handler.codec.DecoderException: javax.net.ssl.SSLHandshakeException: Received fatal alert: bad_certificate at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:480) ~[?:?] at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:279) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?] at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?] at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) ~[?:?] at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) ~[?:?] at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:722) ~[?:?] at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:623) ~[?:?] at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:586) ~[?:?] at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:496) ~[?:?] at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:995) ~[?:?] at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[?:?] at java.lang.Thread.run(Thread.java:833) ~[?:?] Caused by: javax.net.ssl.SSLHandshakeException: Received fatal alert: bad_certificate at sun.security.ssl.Alert.createSSLException(Alert.java:131) ~[?:?] at sun.security.ssl.Alert.createSSLException(Alert.java:117) ~[?:?] at sun.security.ssl.TransportContext.fatal(TransportContext.java:358) ~[?:?] at sun.security.ssl.Alert$AlertConsumer.consume(Alert.java:293) ~[?:?] at sun.security.ssl.TransportContext.dispatch(TransportContext.java:204) ~[?:?] at sun.security.ssl.SSLTransport.decode(SSLTransport.java:172) ~[?:?] at sun.security.ssl.SSLEngineImpl.decode(SSLEngineImpl.java:736) ~[?:?] at sun.security.ssl.SSLEngineImpl.readRecord(SSLEngineImpl.java:691) ~[?:?] at sun.security.ssl.SSLEngineImpl.unwrap(SSLEngineImpl.java:506) ~[?:?] at sun.security.ssl.SSLEngineImpl.unwrap(SSLEngineImpl.java:482) ~[?:?] at javax.net.ssl.SSLEngine.unwrap(SSLEngine.java:679) ~[?:?] at io.netty.handler.ssl.SslHandler$SslEngineType$3.unwrap(SslHandler.java:295) ~[?:?] at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1342) ~[?:?] at io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1235) ~[?:?] at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1284) ~[?:?] at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:510) ~[?:?] at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:449) ~[?:?] ... 16 more [2023-01-06T15:29:37,078][WARN ][o.e.x.s.t.n.SecurityNetty4HttpServerTransport] [audit2] received plaintext http traffic on an https channel, closing connection Netty4HttpChannel{localAddress=/10.16.106.196:9200, remoteAddress=/10.16.24.98:55590} [2023-01-06T15:29:53,850][WARN ][o.e.x.s.t.n.SecurityNetty4HttpServerTransport] [audit2] received plaintext http traffic on an https channel, closing connection Netty4HttpChannel{localAddress=/10.16.106.196:9200, remoteAddress=/10.16.234.71:33520} [2023-01-06T15:30:03,813][WARN ][o.e.x.s.t.n.SecurityNetty4HttpServerTransport] [audit2] received plaintext http traffic on an https channel, closing connection Netty4HttpChannel{localAddress=/10.16.106.196:9200, remoteAddress=/10.16.234.71:33534} [2023-01-06T15:30:18,869][WARN ][o.e.h.AbstractHttpServerTransport] [audit2] caught exception while handling client http traffic, closing connection Netty4HttpChannel{localAddress=/10.16.106.196:9200, remoteAddress=/10.16.234.71:33640} io.netty.handler.codec.DecoderException: javax.net.ssl.SSLHandshakeException: Received fatal alert: bad_certificate at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:480) ~[?:?] at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:279) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?] at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?] at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) ~[?:?] at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) ~[?:?] at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:722) ~[?:?] at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:623) ~[?:?] at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:586) ~[?:?] at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:496) ~[?:?] at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:995) ~[?:?] at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[?:?] at java.lang.Thread.run(Thread.java:833) ~[?:?] Caused by: javax.net.ssl.SSLHandshakeException: Received fatal alert: bad_certificate at sun.security.ssl.Alert.createSSLException(Alert.java:131) ~[?:?] at sun.security.ssl.Alert.createSSLException(Alert.java:117) ~[?:?] at sun.security.ssl.TransportContext.fatal(TransportContext.java:358) ~[?:?] at sun.security.ssl.Alert$AlertConsumer.consume(Alert.java:293) ~[?:?] at sun.security.ssl.TransportContext.dispatch(TransportContext.java:204) ~[?:?] at sun.security.ssl.SSLTransport.decode(SSLTransport.java:172) ~[?:?] at sun.security.ssl.SSLEngineImpl.decode(SSLEngineImpl.java:736) ~[?:?] at sun.security.ssl.SSLEngineImpl.readRecord(SSLEngineImpl.java:691) ~[?:?] at sun.security.ssl.SSLEngineImpl.unwrap(SSLEngineImpl.java:506) ~[?:?] at sun.security.ssl.SSLEngineImpl.unwrap(SSLEngineImpl.java:482) ~[?:?] at javax.net.ssl.SSLEngine.unwrap(SSLEngine.java:679) ~[?:?] at io.netty.handler.ssl.SslHandler$SslEngineType$3.unwrap(SslHandler.java:295) ~[?:?] at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1342) ~[?:?] at io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1235) ~[?:?] at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1284) ~[?:?] at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:510) ~[?:?] at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:449) ~[?:?] ... 16 more [2023-01-06T15:30:21,820][WARN ][o.e.x.s.t.n.SecurityNetty4HttpServerTransport] [audit2] received plaintext http traffic on an https channel, closing connection Netty4HttpChannel{localAddress=/10.16.106.196:9200, remoteAddress=/10.16.24.98:56542} [2023-01-06T15:30:28,893][WARN ][o.e.x.s.t.n.SecurityNetty4HttpServerTransport] [audit2] received plaintext http traffic on an https channel, closing connection Netty4HttpChannel{localAddress=/10.16.106.196:9200, remoteAddress=/10.16.234.71:33654} [2023-01-06T15:30:50,553][WARN ][o.e.x.s.t.n.SecurityNetty4HttpServerTransport] [audit2] received plaintext http traffic on an https channel, closing connection Netty4HttpChannel{localAddress=/10.16.106.196:9200, remoteAddress=/10.16.234.71:33754} [2023-01-06T15:30:52,440][WARN ][o.e.h.AbstractHttpServerTransport] [audit2] caught exception while handling client http traffic, closing connection Netty4HttpChannel{localAddress=/10.16.106.196:9200, remoteAddress=/10.16.234.71:33756} io.netty.handler.codec.DecoderException: javax.net.ssl.SSLHandshakeException: Received fatal alert: bad_certificate at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:480) ~[?:?] at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:279) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?] at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?] at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) ~[?:?] at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) ~[?:?] at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:722) ~[?:?] at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:623) ~[?:?] at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:586) ~[?:?] at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:496) ~[?:?] at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:995) ~[?:?] at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[?:?] at java.lang.Thread.run(Thread.java:833) ~[?:?] Caused by: javax.net.ssl.SSLHandshakeException: Received fatal alert: bad_certificate at sun.security.ssl.Alert.createSSLException(Alert.java:131) ~[?:?] at sun.security.ssl.Alert.createSSLException(Alert.java:117) ~[?:?] at sun.security.ssl.TransportContext.fatal(TransportContext.java:358) ~[?:?] at sun.security.ssl.Alert$AlertConsumer.consume(Alert.java:293) ~[?:?] at sun.security.ssl.TransportContext.dispatch(TransportContext.java:204) ~[?:?] at sun.security.ssl.SSLTransport.decode(SSLTransport.java:172) ~[?:?] at sun.security.ssl.SSLEngineImpl.decode(SSLEngineImpl.java:736) ~[?:?] at sun.security.ssl.SSLEngineImpl.readRecord(SSLEngineImpl.java:691) ~[?:?] at sun.security.ssl.SSLEngineImpl.unwrap(SSLEngineImpl.java:506) ~[?:?] at sun.security.ssl.SSLEngineImpl.unwrap(SSLEngineImpl.java:482) ~[?:?] at javax.net.ssl.SSLEngine.unwrap(SSLEngine.java:679) ~[?:?] at io.netty.handler.ssl.SslHandler$SslEngineType$3.unwrap(SslHandler.java:295) ~[?:?] at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1342) ~[?:?] at io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1235) ~[?:?] at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1284) ~[?:?] at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:510) ~[?:?] at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:449) ~[?:?] ... 16 more [2023-01-06T15:31:05,342][WARN ][o.e.x.s.t.n.SecurityNetty4HttpServerTransport] [audit2] received plaintext http traffic on an https channel, closing connection Netty4HttpChannel{localAddress=/10.16.106.196:9200, remoteAddress=/10.16.234.71:33774} [2023-01-06T15:31:07,615][WARN ][o.e.x.s.t.n.SecurityNetty4HttpServerTransport] [audit2] received plaintext http traffic on an https channel, closing connection Netty4HttpChannel{localAddress=/10.16.106.196:9200, remoteAddress=/10.16.24.98:57232} [2023-01-06T15:31:41,314][WARN ][o.e.x.s.t.n.SecurityNetty4HttpServerTransport] [audit2] received plaintext http traffic on an https channel, closing connection Netty4HttpChannel{localAddress=/10.16.106.196:9200, remoteAddress=/10.16.234.71:33948} [2023-01-06T15:31:49,012][WARN ][o.e.x.s.t.n.SecurityNetty4HttpServerTransport] [audit2] received plaintext http traffic on an https channel, closing connection Netty4HttpChannel{localAddress=/10.16.106.196:9200, remoteAddress=/10.16.234.71:33968} [2023-01-06T15:31:50,165][WARN ][o.e.x.s.t.n.SecurityNetty4HttpServerTransport] [audit2] received plaintext http traffic on an https channel, closing connection Netty4HttpChannel{localAddress=/10.16.106.196:9200, remoteAddress=/10.16.24.98:57920} [2023-01-06T15:31:50,811][WARN ][o.e.h.AbstractHttpServerTransport] [audit2] caught exception while handling client http traffic, closing connection Netty4HttpChannel{localAddress=/10.16.106.196:9200, remoteAddress=/10.16.234.71:33974} io.netty.handler.codec.DecoderException: javax.net.ssl.SSLHandshakeException: Received fatal alert: bad_certificate at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:480) ~[?:?] at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:279) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?] at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?] at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) ~[?:?] at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) ~[?:?] at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:722) ~[?:?] at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:623) ~[?:?] at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:586) ~[?:?] at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:496) ~[?:?] at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:995) ~[?:?] at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[?:?] at java.lang.Thread.run(Thread.java:833) ~[?:?] Caused by: javax.net.ssl.SSLHandshakeException: Received fatal alert: bad_certificate at sun.security.ssl.Alert.createSSLException(Alert.java:131) ~[?:?] at sun.security.ssl.Alert.createSSLException(Alert.java:117) ~[?:?] at sun.security.ssl.TransportContext.fatal(TransportContext.java:358) ~[?:?] at sun.security.ssl.Alert$AlertConsumer.consume(Alert.java:293) ~[?:?] at sun.security.ssl.TransportContext.dispatch(TransportContext.java:204) ~[?:?] at sun.security.ssl.SSLTransport.decode(SSLTransport.java:172) ~[?:?] at sun.security.ssl.SSLEngineImpl.decode(SSLEngineImpl.java:736) ~[?:?] at sun.security.ssl.SSLEngineImpl.readRecord(SSLEngineImpl.java:691) ~[?:?] at sun.security.ssl.SSLEngineImpl.unwrap(SSLEngineImpl.java:506) ~[?:?] at sun.security.ssl.SSLEngineImpl.unwrap(SSLEngineImpl.java:482) ~[?:?] at javax.net.ssl.SSLEngine.unwrap(SSLEngine.java:679) ~[?:?] at io.netty.handler.ssl.SslHandler$SslEngineType$3.unwrap(SslHandler.java:295) ~[?:?] at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1342) ~[?:?] at io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1235) ~[?:?] at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1284) ~[?:?] at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:510) ~[?:?] at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessaePreformatted text[?:?]``` ... 16 more
type or paste code here
```There is nothing in this logs that may indicate an issue with your cluster.
However you have logs that tells that something is trying to connect to your cluster using http instead of https.
Like this:
[2023-01-06T15:31:50,811][WARN ][o.e.h.AbstractHttpServerTransport] [audit2] caught exception while handling client http traffic, closing connection Netty4HttpChannel{localAddress=/10.16.106.196:9200, remoteAddress=/10.16.234.71:33974} io.netty.handler.codec.DecoderException: javax.net.ssl.SSLHandshakeException: Received fatal alert: bad_certificate
Do you have any ERROR lines in your log? These would indicate some issue in your cluster, the lines you shared are all WARN lines about some client trying to connect using http instead of https.
How did you identify that your cluster was not working while the audit1 server was down?
What is the result of curl to your audit2 and audit3 server when the audit1 is down?
1 Like
I don't see any 'error' lines in any log file.
I determined that my cluster was not working when I attempted to log into the kibana web pages but they did not even open up.
Servers don't respond at all with the curl commands after shutdown of audit1.
I think that the problem is with the node certs. When I run 'GET /_ssl/certificates' in the console, only audit1 has the correct hostname info for the subject_dn. I've been running thru the documented 'update security certificates' on the web, but whenever I try to list the existing certs in the *.p12 files, I get an 'invalid keystore format' error. This also occurs when I attempt to import a new cert into the keystore.
What am I doing wrong?
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.