# Three node cluster failes completely when one node shuts down **URL:** \ **Category:** Elasticsearch\ **Created:** [January 6, 2023, 8:16pm UTC](https://discuss.elastic.co/t/three-node-cluster-failes-completely-when-one-node-shuts-down/322638 "2023-01-06T20:16:53Z")\ **Posts on this page:** 11\ **Page:** 1
**Author:** ![cdy5159](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/cdy5159/32/115812_2.png) [@cdy5159](https://discuss.elastic.co/u/cdy5159)\ **Post date:** [January 6, 2023, 8:16pm UTC](https://discuss.elastic.co/t/three-node-cluster-failes-completely-when-one-node-shuts-down/322638/1 "2023-01-06T20:16:53Z")
I have a three node cluster with Elastic 8.4.2. When all three nodes are up, it all appears fine. I built the cluster with a single node (audit1), but later added the other two nodes (audit2 & audit3). If I shutdown either audit2 or audit3, the cluster keeps on running fine with no observable issues. However, if I shutdown audit1, the entire cluster is no longer accessible. From other posts, I've looked at voting exclusions and none are there. I've made these three nodes as identical as I know how, but there appears to be some other thing that makes audit1 unique. Anyone have any ideas? ---
**Author:** ![leandrojmp](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/leandrojmp/32/107231_2.png) [@leandrojmp](https://discuss.elastic.co/u/leandrojmp)\ **Post date:** [January 6, 2023, 8:23pm UTC](https://discuss.elastic.co/t/three-node-cluster-failes-completely-when-one-node-shuts-down/322638/2 "2023-01-06T20:23:56Z")
Please share the `elasticsearch.yml` of each node. ---
**Author:** ![cdy5159](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/cdy5159/32/115812_2.png) [@cdy5159](https://discuss.elastic.co/u/cdy5159)\ **Post date:** [January 6, 2023, 9:11pm UTC](https://discuss.elastic.co/t/three-node-cluster-failes-completely-when-one-node-shuts-down/322638/3 "2023-01-06T21:11:04Z")
cluster.name: UnclassTest node.name: audit1 path.data: /var/lib/elasticsearch path.logs: /var/log/elasticsearch network.host: 0.0.0.0 http.port: 9200 discovery.seed\_hosts: - "audit-dev.u.l3" - "auditsvr2.u.l3" - "auditsvr3.u.l3" cluster.initial\_master\_nodes: - "audit-dev.u.l3" - "auditsvr2.u.l3" - "auditsvr3.u.l3" node.roles: ["master","data","ingest"] xpack.security.enabled: true xpack.security.enrollment.enabled: true xpack.security.http.ssl: enabled: true keystore.path: certs/http.p12 xpack.security.transport.ssl: enabled: true verification\_mode: certificate keystore.path: certs/transport.p12 truststore.path: certs/transport.p12 http.host: audit-dev.u.l3 transport.host: audit-dev.u.l3 The other nodes have identical yml files with the exception of the node.name, http.host, and transport.host. Those fields are specific to the node's hostname. ---
**Author:** ![leandrojmp](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/leandrojmp/32/107231_2.png) [@leandrojmp](https://discuss.elastic.co/u/leandrojmp)\ **Post date:** [January 6, 2023, 9:13pm UTC](https://discuss.elastic.co/t/three-node-cluster-failes-completely-when-one-node-shuts-down/322638/4 "2023-01-06T21:13:59Z")
Is `audit-dev` your `audit1` node? What do you have in the logs fo the other nodes when you shutdown this node? Also, share your configuration and logs using the preformatted option, the `` button in the editor, it makes easier to read the configuration. ---
**Author:** ![cdy5159](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/cdy5159/32/115812_2.png) [@cdy5159](https://discuss.elastic.co/u/cdy5159)\ **Post date:** [January 6, 2023, 9:20pm UTC](https://discuss.elastic.co/t/three-node-cluster-failes-completely-when-one-node-shuts-down/322638/5 "2023-01-06T21:20:26Z")
elasticsearch received SSLhandshake exception: bad\_certificate. Surviving nodes are showing high disk watermark errors, but system df -h commands show no such problem. Yes, audit-dev is audit1. ---
**Author:** ![leandrojmp](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/leandrojmp/32/107231_2.png) [@leandrojmp](https://discuss.elastic.co/u/leandrojmp)\ **Post date:** [January 6, 2023, 9:26pm UTC](https://discuss.elastic.co/t/three-node-cluster-failes-completely-when-one-node-shuts-down/322638/6 "2023-01-06T21:26:38Z")
> [@cdy5159](#): > > elasticsearch received SSLhandshake exception: bad\_certificate. Where? From Which node? if you have this error you wouldn't have formed a cluster. > [@cdy5159](#): > > Surviving nodes are showing high disk watermark errors, but system df -h commands show no such problem. How much used and free space do you have in the elasticsearch data path? You need to share the logs, without seeing the logs it is not possible to know what may be the issue. ---
**Author:** ![cdy5159](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/cdy5159/32/115812_2.png) [@cdy5159](https://discuss.elastic.co/u/cdy5159)\ **Post date:** [January 6, 2023, 9:42pm UTC](https://discuss.elastic.co/t/three-node-cluster-failes-completely-when-one-node-shuts-down/322638/7 "2023-01-06T21:42:04Z")
This is the elasticsearch log contents from audit2 when audit1 was powered off. ````Preformatted text`[2023-01-06T15:26:03,943][WARN][o.e.x.s.t.n.SecurityNetty4HttpServerTransport] [audit2] received plaintext http traffic on an https channel, closing connection Netty4HttpChannel{localAddress=/10.16.106.196:9200, remoteAddress=/10.16.234.71:60744} [2023-01-06T15:26:08,366][WARN][o.e.x.s.t.n.SecurityNetty4HttpServerTransport] [audit2] received plaintext http traffic on an https channel, closing connection Netty4HttpChannel{localAddress=/10.16.106.196:9200, remoteAddress=/10.16.24.98:52260} [2023-01-06T15:26:25,197][WARN][o.e.x.s.t.n.SecurityNetty4HttpServerTransport] [audit2] received plaintext http traffic on an https channel, closing connection Netty4HttpChannel{localAddress=/10.16.106.196:9200, remoteAddress=/10.16.234.71:60862} [2023-01-06T15:26:34,351][WARN][o.e.h.AbstractHttpServerTransport] [audit2] caught exception while handling client http traffic, closing connection Netty4HttpChannel{localAddress=/10.16.106.196:9200, remoteAddress=/10.16.234.71:60868} io.netty.handler.codec.DecoderException: javax.net.ssl.SSLHandshakeException: Received fatal alert: bad_certificate at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:480) ~[?:?] at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:279) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?] at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?] at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) ~[?:?] at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) ~[?:?] at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:722) ~[?:?] at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:623) ~[?:?] at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:586) ~[?:?] at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:496) ~[?:?] at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:995) ~[?:?] at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[?:?] at java.lang.Thread.run(Thread.java:833) ~[?:?] Caused by: javax.net.ssl.SSLHandshakeException: Received fatal alert: bad_certificate at sun.security.ssl.Alert.createSSLException(Alert.java:131) ~[?:?] at sun.security.ssl.Alert.createSSLException(Alert.java:117) ~[?:?] at sun.security.ssl.TransportContext.fatal(TransportContext.java:358) ~[?:?] at sun.security.ssl.Alert$AlertConsumer.consume(Alert.java:293) ~[?:?] at sun.security.ssl.TransportContext.dispatch(TransportContext.java:204) ~[?:?] at sun.security.ssl.SSLTransport.decode(SSLTransport.java:172) ~[?:?] at sun.security.ssl.SSLEngineImpl.decode(SSLEngineImpl.java:736) ~[?:?] at sun.security.ssl.SSLEngineImpl.readRecord(SSLEngineImpl.java:691) ~[?:?] at sun.security.ssl.SSLEngineImpl.unwrap(SSLEngineImpl.java:506) ~[?:?] at sun.security.ssl.SSLEngineImpl.unwrap(SSLEngineImpl.java:482) ~[?:?] at javax.net.ssl.SSLEngine.unwrap(SSLEngine.java:679) ~[?:?] at io.netty.handler.ssl.SslHandler$SslEngineType$3.unwrap(SslHandler.java:295) ~[?:?] at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1342) ~[?:?] at io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1235) ~[?:?] at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1284) ~[?:?] at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:510) ~[?:?] at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:449) ~[?:?] ... 16 more [2023-01-06T15:26:47,713][WARN][o.e.x.s.t.n.SecurityNetty4HttpServerTransport] [audit2] received plaintext http traffic on an https channel, closing connection Netty4HttpChannel{localAddress=/10.16.106.196:9200, remoteAddress=/10.16.24.98:52858} [2023-01-06T15:26:59,066][WARN][o.e.x.s.t.n.SecurityNetty4HttpServerTransport] [audit2] received plaintext http traffic on an https channel, closing connection Netty4HttpChannel{localAddress=/10.16.106.196:9200, remoteAddress=/10.16.234.71:60948} [2023-01-06T15:27:15,191][WARN][o.e.x.s.t.n.SecurityNetty4HttpServerTransport] [audit2] received plaintext http traffic on an https channel, closing connection Netty4HttpChannel{localAddress=/10.16.106.196:9200, remoteAddress=/10.16.234.71:32962} [2023-01-06T15:27:19,307][WARN][o.e.h.AbstractHttpServerTransport] [audit2] caught exception while handling client http traffic, closing connection Netty4HttpChannel{localAddress=/10.16.106.196:9200, remoteAddress=/10.16.234.71:32968} io.netty.handler.codec.DecoderException: javax.net.ssl.SSLHandshakeException: Received fatal alert: bad_certificate at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:480) ~[?:?] at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:279) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?] at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?] at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) ~[?:?] at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) ~[?:?] at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:722) ~[?:?] at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:623) ~[?:?] at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:586) ~[?:?] at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:496) ~[?:?] at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:995) ~[?:?] at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[?:?] at java.lang.Thread.run(Thread.java:833) ~[?:?] Caused by: javax.net.ssl.SSLHandshakeException: Received fatal alert: bad_certificate at sun.security.ssl.Alert.createSSLException(Alert.java:131) ~[?:?] at sun.security.ssl.Alert.createSSLException(Alert.java:117) ~[?:?] at sun.security.ssl.TransportContext.fatal(TransportContext.java:358) ~[?:?] at sun.security.ssl.Alert$AlertConsumer.consume(Alert.java:293) ~[?:?] at sun.security.ssl.TransportContext.dispatch(TransportContext.java:204) ~[?:?] at sun.security.ssl.SSLTransport.decode(SSLTransport.java:172) ~[?:?] at sun.security.ssl.SSLEngineImpl.decode(SSLEngineImpl.java:736) ~[?:?] at sun.security.ssl.SSLEngineImpl.readRecord(SSLEngineImpl.java:691) ~[?:?] at sun.security.ssl.SSLEngineImpl.unwrap(SSLEngineImpl.java:506) ~[?:?] at sun.security.ssl.SSLEngineImpl.unwrap(SSLEngineImpl.java:482) ~[?:?] at javax.net.ssl.SSLEngine.unwrap(SSLEngine.java:679) ~[?:?] at io.netty.handler.ssl.SslHandler$SslEngineType$3.unwrap(SslHandler.java:295) ~[?:?] at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1342) ~[?:?] at io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1235) ~[?:?] at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1284) ~[?:?] at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:510) ~[?:?] at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:449) ~[?:?] ... 16 more [2023-01-06T15:27:34,387][WARN][o.e.x.s.t.n.SecurityNetty4HttpServerTransport] [audit2] received plaintext http traffic on an https channel, closing connection Netty4HttpChannel{localAddress=/10.16.106.196:9200, remoteAddress=/10.16.24.98:53608} [2023-01-06T15:27:44,875][WARN][o.e.x.s.t.n.SecurityNetty4HttpServerTransport] [audit2] received plaintext http traffic on an https channel, closing connection Netty4HttpChannel{localAddress=/10.16.106.196:9200, remoteAddress=/10.16.234.71:33066} [2023-01-06T15:27:52,937][WARN][o.e.h.AbstractHttpServerTransport] [audit2] caught exception while handling client http traffic, closing connection Netty4HttpChannel{localAddress=/10.16.106.196:9200, remoteAddress=/10.16.234.71:33072} io.netty.handler.codec.DecoderException: javax.net.ssl.SSLHandshakeException: Received fatal alert: bad_certificate at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:480) ~[?:?] at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:279) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?] at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?] at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) ~[?:?] at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) ~[?:?] at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:722) ~[?:?] at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:623) ~[?:?] at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:586) ~[?:?] at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:496) ~[?:?] at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:995) ~[?:?] at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[?:?] at java.lang.Thread.run(Thread.java:833) ~[?:?] Caused by: javax.net.ssl.SSLHandshakeException: Received fatal alert: bad_certificate at sun.security.ssl.Alert.createSSLException(Alert.java:131) ~[?:?] at sun.security.ssl.Alert.createSSLException(Alert.java:117) ~[?:?] at sun.security.ssl.TransportContext.fatal(TransportContext.java:358) ~[?:?] at sun.security.ssl.Alert$AlertConsumer.consume(Alert.java:293) ~[?:?] at sun.security.ssl.TransportContext.dispatch(TransportContext.java:204) ~[?:?] at sun.security.ssl.SSLTransport.decode(SSLTransport.java:172) ~[?:?] at sun.security.ssl.SSLEngineImpl.decode(SSLEngineImpl.java:736) ~[?:?] at sun.security.ssl.SSLEngineImpl.readRecord(SSLEngineImpl.java:691) ~[?:?] at sun.security.ssl.SSLEngineImpl.unwrap(SSLEngineImpl.java:506) ~[?:?] at sun.security.ssl.SSLEngineImpl.unwrap(SSLEngineImpl.java:482) ~[?:?] at javax.net.ssl.SSLEngine.unwrap(SSLEngine.java:679) ~[?:?] at io.netty.handler.ssl.SslHandler$SslEngineType$3.unwrap(SslHandler.java:295) ~[?:?] at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1342) ~[?:?] at io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1235) ~[?:?] at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1284) ~[?:?] at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:510) ~[?:?] at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:449) ~[?:?] ... 16 more [2023-01-06T15:28:02,792][WARN][o.e.x.s.t.n.SecurityNetty4HttpServerTransport] [audit2] received plaintext http traffic on an https channel, closing connection Netty4HttpChannel{localAddress=/10.16.106.196:9200, remoteAddress=/10.16.234.71:33092} [2023-01-06T15:28:12,650][WARN][o.e.x.s.t.n.SecurityNetty4HttpServerTransport] [audit2] received plaintext http traffic on an https channel, closing connection Netty4HttpChannel{localAddress=/10.16.106.196:9200, remoteAddress=/10.16.24.98:54194} [2023-01-06T15:28:26,435][WARN][o.e.x.s.t.n.SecurityNetty4HttpServerTransport] [audit2] received plaintext http traffic on an https channel, closing connection Netty4HttpChannel{localAddress=/10.16.106.196:9200, remoteAddress=/10.16.234.71:33214} [2023-01-06T15:28:33,638][WARN][o.e.x.s.t.n.SecurityNetty4HttpServerTransport] [audit2] received plaintext http traffic on an https channel, closing connection Netty4HttpChannel{localAddress=/10.16.106.196:9200, remoteAddress=/10.16.234.71:33222} [2023-01-06T15:28:51,169][WARN][o.e.h.AbstractHttpServerTransport] [audit2] caught exception while handling client http traffic, closing connection Netty4HttpChannel{localAddress=/10.16.106.196:9200, remoteAddress=/10.16.234.71:33290} io.netty.handler.codec.DecoderException: javax.net.ssl.SSLHandshakeException: Received fatal alert: bad_certificate at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:480) ~[?:?] at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:279) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?] at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?] at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) ~[?:?] at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) ~[?:?] at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:722) ~[?:?] at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:623) ~[?:?] at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:586) ~[?:?] at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:496) ~[?:?] at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:995) ~[?:?] at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[?:?] at java.lang.Thread.run(Thread.java:833) ~[?:?] Caused by: javax.net.ssl.SSLHandshakeException: Received fatal alert: bad_certificate at sun.security.ssl.Alert.createSSLException(Alert.java:131) ~[?:?] at sun.security.ssl.Alert.createSSLException(Alert.java:117) ~[?:?] at sun.security.ssl.TransportContext.fatal(TransportContext.java:358) ~[?:?] at sun.security.ssl.Alert$AlertConsumer.consume(Alert.java:293) ~[?:?] at sun.security.ssl.TransportContext.dispatch(TransportContext.java:204) ~[?:?] at sun.security.ssl.SSLTransport.decode(SSLTransport.java:172) ~[?:?] at sun.security.ssl.SSLEngineImpl.decode(SSLEngineImpl.java:736) ~[?:?] at sun.security.ssl.SSLEngineImpl.readRecord(SSLEngineImpl.java:691) ~[?:?] at sun.security.ssl.SSLEngineImpl.unwrap(SSLEngineImpl.java:506) ~[?:?] at sun.security.ssl.SSLEngineImpl.unwrap(SSLEngineImpl.java:482) ~[?:?] at javax.net.ssl.SSLEngine.unwrap(SSLEngine.java:679) ~[?:?] at io.netty.handler.ssl.SslHandler$SslEngineType$3.unwrap(SslHandler.java:295) ~[?:?] at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1342) ~[?:?] at io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1235) ~[?:?] at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1284) ~[?:?] at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:510) ~[?:?] at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:449) ~[?:?] ... 16 more [2023-01-06T15:29:01,469][WARN][o.e.x.s.t.n.SecurityNetty4HttpServerTransport] [audit2] received plaintext http traffic on an https channel, closing connection Netty4HttpChannel{localAddress=/10.16.106.196:9200, remoteAddress=/10.16.24.98:54966} [2023-01-06T15:29:09,661][WARN][o.e.x.s.t.n.SecurityNetty4HttpServerTransport] [audit2] received plaintext http traffic on an https channel, closing connection Netty4HttpChannel{localAddress=/10.16.106.196:9200, remoteAddress=/10.16.234.71:33356} [2023-01-06T15:29:19,479][WARN][o.e.x.s.t.n.SecurityNetty4HttpServerTransport] [audit2] received plaintext http traffic on an https channel, closing connection Netty4HttpChannel{localAddress=/10.16.106.196:9200, remoteAddress=/10.16.234.71:33416} [2023-01-06T15:29:21,832][WARN][o.e.h.AbstractHttpServerTransport] [audit2] caught exception while handling client http traffic, closing connection Netty4HttpChannel{localAddress=/10.16.106.196:9200, remoteAddress=/10.16.234.71:33418} io.netty.handler.codec.DecoderException: javax.net.ssl.SSLHandshakeException: Received fatal alert: bad_certificate at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:480) ~[?:?] at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:279) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?] at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?] at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) ~[?:?] at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) ~[?:?] at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:722) ~[?:?] at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:623) ~[?:?] at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:586) ~[?:?] at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:496) ~[?:?] at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:995) ~[?:?] at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[?:?] at java.lang.Thread.run(Thread.java:833) ~[?:?] Caused by: javax.net.ssl.SSLHandshakeException: Received fatal alert: bad_certificate at sun.security.ssl.Alert.createSSLException(Alert.java:131) ~[?:?] at sun.security.ssl.Alert.createSSLException(Alert.java:117) ~[?:?] at sun.security.ssl.TransportContext.fatal(TransportContext.java:358) ~[?:?] at sun.security.ssl.Alert$AlertConsumer.consume(Alert.java:293) ~[?:?] at sun.security.ssl.TransportContext.dispatch(TransportContext.java:204) ~[?:?] at sun.security.ssl.SSLTransport.decode(SSLTransport.java:172) ~[?:?] at sun.security.ssl.SSLEngineImpl.decode(SSLEngineImpl.java:736) ~[?:?] at sun.security.ssl.SSLEngineImpl.readRecord(SSLEngineImpl.java:691) ~[?:?] at sun.security.ssl.SSLEngineImpl.unwrap(SSLEngineImpl.java:506) ~[?:?] at sun.security.ssl.SSLEngineImpl.unwrap(SSLEngineImpl.java:482) ~[?:?] at javax.net.ssl.SSLEngine.unwrap(SSLEngine.java:679) ~[?:?] at io.netty.handler.ssl.SslHandler$SslEngineType$3.unwrap(SslHandler.java:295) ~[?:?] at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1342) ~[?:?] at io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1235) ~[?:?] at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1284) ~[?:?] at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:510) ~[?:?] at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:449) ~[?:?] ... 16 more [2023-01-06T15:29:37,078][WARN][o.e.x.s.t.n.SecurityNetty4HttpServerTransport] [audit2] received plaintext http traffic on an https channel, closing connection Netty4HttpChannel{localAddress=/10.16.106.196:9200, remoteAddress=/10.16.24.98:55590} [2023-01-06T15:29:53,850][WARN][o.e.x.s.t.n.SecurityNetty4HttpServerTransport] [audit2] received plaintext http traffic on an https channel, closing connection Netty4HttpChannel{localAddress=/10.16.106.196:9200, remoteAddress=/10.16.234.71:33520} [2023-01-06T15:30:03,813][WARN][o.e.x.s.t.n.SecurityNetty4HttpServerTransport] [audit2] received plaintext http traffic on an https channel, closing connection Netty4HttpChannel{localAddress=/10.16.106.196:9200, remoteAddress=/10.16.234.71:33534} [2023-01-06T15:30:18,869][WARN][o.e.h.AbstractHttpServerTransport] [audit2] caught exception while handling client http traffic, closing connection Netty4HttpChannel{localAddress=/10.16.106.196:9200, remoteAddress=/10.16.234.71:33640} io.netty.handler.codec.DecoderException: javax.net.ssl.SSLHandshakeException: Received fatal alert: bad_certificate at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:480) ~[?:?] at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:279) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?] at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?] at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) ~[?:?] at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) ~[?:?] at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:722) ~[?:?] at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:623) ~[?:?] at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:586) ~[?:?] at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:496) ~[?:?] at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:995) ~[?:?] at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[?:?] at java.lang.Thread.run(Thread.java:833) ~[?:?] Caused by: javax.net.ssl.SSLHandshakeException: Received fatal alert: bad_certificate at sun.security.ssl.Alert.createSSLException(Alert.java:131) ~[?:?] at sun.security.ssl.Alert.createSSLException(Alert.java:117) ~[?:?] at sun.security.ssl.TransportContext.fatal(TransportContext.java:358) ~[?:?] at sun.security.ssl.Alert$AlertConsumer.consume(Alert.java:293) ~[?:?] at sun.security.ssl.TransportContext.dispatch(TransportContext.java:204) ~[?:?] at sun.security.ssl.SSLTransport.decode(SSLTransport.java:172) ~[?:?] at sun.security.ssl.SSLEngineImpl.decode(SSLEngineImpl.java:736) ~[?:?] at sun.security.ssl.SSLEngineImpl.readRecord(SSLEngineImpl.java:691) ~[?:?] at sun.security.ssl.SSLEngineImpl.unwrap(SSLEngineImpl.java:506) ~[?:?] at sun.security.ssl.SSLEngineImpl.unwrap(SSLEngineImpl.java:482) ~[?:?] at javax.net.ssl.SSLEngine.unwrap(SSLEngine.java:679) ~[?:?] at io.netty.handler.ssl.SslHandler$SslEngineType$3.unwrap(SslHandler.java:295) ~[?:?] at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1342) ~[?:?] at io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1235) ~[?:?] at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1284) ~[?:?] at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:510) ~[?:?] at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:449) ~[?:?] ... 16 more [2023-01-06T15:30:21,820][WARN][o.e.x.s.t.n.SecurityNetty4HttpServerTransport] [audit2] received plaintext http traffic on an https channel, closing connection Netty4HttpChannel{localAddress=/10.16.106.196:9200, remoteAddress=/10.16.24.98:56542} [2023-01-06T15:30:28,893][WARN][o.e.x.s.t.n.SecurityNetty4HttpServerTransport] [audit2] received plaintext http traffic on an https channel, closing connection Netty4HttpChannel{localAddress=/10.16.106.196:9200, remoteAddress=/10.16.234.71:33654} [2023-01-06T15:30:50,553][WARN][o.e.x.s.t.n.SecurityNetty4HttpServerTransport] [audit2] received plaintext http traffic on an https channel, closing connection Netty4HttpChannel{localAddress=/10.16.106.196:9200, remoteAddress=/10.16.234.71:33754} [2023-01-06T15:30:52,440][WARN][o.e.h.AbstractHttpServerTransport] [audit2] caught exception while handling client http traffic, closing connection Netty4HttpChannel{localAddress=/10.16.106.196:9200, remoteAddress=/10.16.234.71:33756} io.netty.handler.codec.DecoderException: javax.net.ssl.SSLHandshakeException: Received fatal alert: bad_certificate at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:480) ~[?:?] at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:279) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?] at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?] at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) ~[?:?] at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) ~[?:?] at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:722) ~[?:?] at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:623) ~[?:?] at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:586) ~[?:?] at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:496) ~[?:?] at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:995) ~[?:?] at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[?:?] at java.lang.Thread.run(Thread.java:833) ~[?:?] Caused by: javax.net.ssl.SSLHandshakeException: Received fatal alert: bad_certificate at sun.security.ssl.Alert.createSSLException(Alert.java:131) ~[?:?] at sun.security.ssl.Alert.createSSLException(Alert.java:117) ~[?:?] at sun.security.ssl.TransportContext.fatal(TransportContext.java:358) ~[?:?] at sun.security.ssl.Alert$AlertConsumer.consume(Alert.java:293) ~[?:?] at sun.security.ssl.TransportContext.dispatch(TransportContext.java:204) ~[?:?] at sun.security.ssl.SSLTransport.decode(SSLTransport.java:172) ~[?:?] at sun.security.ssl.SSLEngineImpl.decode(SSLEngineImpl.java:736) ~[?:?] at sun.security.ssl.SSLEngineImpl.readRecord(SSLEngineImpl.java:691) ~[?:?] at sun.security.ssl.SSLEngineImpl.unwrap(SSLEngineImpl.java:506) ~[?:?] at sun.security.ssl.SSLEngineImpl.unwrap(SSLEngineImpl.java:482) ~[?:?] at javax.net.ssl.SSLEngine.unwrap(SSLEngine.java:679) ~[?:?] at io.netty.handler.ssl.SslHandler$SslEngineType$3.unwrap(SslHandler.java:295) ~[?:?] at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1342) ~[?:?] at io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1235) ~[?:?] at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1284) ~[?:?] at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:510) ~[?:?] at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:449) ~[?:?] ... 16 more [2023-01-06T15:31:05,342][WARN][o.e.x.s.t.n.SecurityNetty4HttpServerTransport] [audit2] received plaintext http traffic on an https channel, closing connection Netty4HttpChannel{localAddress=/10.16.106.196:9200, remoteAddress=/10.16.234.71:33774} [2023-01-06T15:31:07,615][WARN][o.e.x.s.t.n.SecurityNetty4HttpServerTransport] [audit2] received plaintext http traffic on an https channel, closing connection Netty4HttpChannel{localAddress=/10.16.106.196:9200, remoteAddress=/10.16.24.98:57232} [2023-01-06T15:31:41,314][WARN][o.e.x.s.t.n.SecurityNetty4HttpServerTransport] [audit2] received plaintext http traffic on an https channel, closing connection Netty4HttpChannel{localAddress=/10.16.106.196:9200, remoteAddress=/10.16.234.71:33948} [2023-01-06T15:31:49,012][WARN][o.e.x.s.t.n.SecurityNetty4HttpServerTransport] [audit2] received plaintext http traffic on an https channel, closing connection Netty4HttpChannel{localAddress=/10.16.106.196:9200, remoteAddress=/10.16.234.71:33968} [2023-01-06T15:31:50,165][WARN][o.e.x.s.t.n.SecurityNetty4HttpServerTransport] [audit2] received plaintext http traffic on an https channel, closing connection Netty4HttpChannel{localAddress=/10.16.106.196:9200, remoteAddress=/10.16.24.98:57920} [2023-01-06T15:31:50,811][WARN][o.e.h.AbstractHttpServerTransport] [audit2] caught exception while handling client http traffic, closing connection Netty4HttpChannel{localAddress=/10.16.106.196:9200, remoteAddress=/10.16.234.71:33974} io.netty.handler.codec.DecoderException: javax.net.ssl.SSLHandshakeException: Received fatal alert: bad_certificate at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:480) ~[?:?] at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:279) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?] at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?] at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) ~[?:?] at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) ~[?:?] at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:722) ~[?:?] at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:623) ~[?:?] at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:586) ~[?:?] at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:496) ~[?:?] at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:995) ~[?:?] at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[?:?] at java.lang.Thread.run(Thread.java:833) ~[?:?] Caused by: javax.net.ssl.SSLHandshakeException: Received fatal alert: bad_certificate at sun.security.ssl.Alert.createSSLException(Alert.java:131) ~[?:?] at sun.security.ssl.Alert.createSSLException(Alert.java:117) ~[?:?] at sun.security.ssl.TransportContext.fatal(TransportContext.java:358) ~[?:?] at sun.security.ssl.Alert$AlertConsumer.consume(Alert.java:293) ~[?:?] at sun.security.ssl.TransportContext.dispatch(TransportContext.java:204) ~[?:?] at sun.security.ssl.SSLTransport.decode(SSLTransport.java:172) ~[?:?] at sun.security.ssl.SSLEngineImpl.decode(SSLEngineImpl.java:736) ~[?:?] at sun.security.ssl.SSLEngineImpl.readRecord(SSLEngineImpl.java:691) ~[?:?] at sun.security.ssl.SSLEngineImpl.unwrap(SSLEngineImpl.java:506) ~[?:?] at sun.security.ssl.SSLEngineImpl.unwrap(SSLEngineImpl.java:482) ~[?:?] at javax.net.ssl.SSLEngine.unwrap(SSLEngine.java:679) ~[?:?] at io.netty.handler.ssl.SslHandler$SslEngineType$3.unwrap(SslHandler.java:295) ~[?:?] at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1342) ~[?:?] at io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1235) ~[?:?] at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1284) ~[?:?] at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:510) ~[?:?] at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessae`Preformatted text`[?:?]``` ... 16 more` ````auto type or paste code here ``` ```` ---
**Author:** ![leandrojmp](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/leandrojmp/32/107231_2.png) [@leandrojmp](https://discuss.elastic.co/u/leandrojmp)\ **Post date:** [January 7, 2023, 12:37am UTC](https://discuss.elastic.co/t/three-node-cluster-failes-completely-when-one-node-shuts-down/322638/8 "2023-01-07T00:37:50Z")
There is nothing in this logs that may indicate an issue with your cluster. However you have logs that tells that something is trying to connect to your cluster using `http` instead of `https`. Like this: > [2023-01-06T15:31:50,811][WARN][o.e.h.AbstractHttpServerTransport] [audit2] caught exception while handling client http traffic, closing connection Netty4HttpChannel{localAddress=/10.16.106.196:9200, remoteAddress=/10.16.234.71:33974} io.netty.handler.codec.DecoderException: javax.net.ssl.SSLHandshakeException: Received fatal alert: bad\_certificate Do you have any **ERROR** lines in your log? These would indicate some issue in your cluster, the lines you shared are all **WARN** lines about some client trying to connect using `http` instead of `https`. How did you identify that your cluster was not working while the `audit1` server was down? What is the result of curl to your `audit2` and `audit3` server when the `audit1` is down? ---
**Author:** ![cdy5159](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/cdy5159/32/115812_2.png) [@cdy5159](https://discuss.elastic.co/u/cdy5159)\ **Post date:** [January 9, 2023, 1:28pm UTC](https://discuss.elastic.co/t/three-node-cluster-failes-completely-when-one-node-shuts-down/322638/9 "2023-01-09T13:28:11Z")
I don't see any 'error' lines in any log file. I determined that my cluster was not working when I attempted to log into the kibana web pages but they did not even open up. Servers don't respond at all with the curl commands after shutdown of audit1. ---
**Author:** ![cdy5159](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/cdy5159/32/115812_2.png) [@cdy5159](https://discuss.elastic.co/u/cdy5159)\ **Post date:** [January 10, 2023, 4:36pm UTC](https://discuss.elastic.co/t/three-node-cluster-failes-completely-when-one-node-shuts-down/322638/10 "2023-01-10T16:36:01Z")
I think that the problem is with the node certs. When I run 'GET /\_ssl/certificates' in the console, only audit1 has the correct hostname info for the subject\_dn. I've been running thru the documented 'update security certificates' on the web, but whenever I try to list the existing certs in the \*.p12 files, I get an 'invalid keystore format' error. This also occurs when I attempt to import a new cert into the keystore. What am I doing wrong? ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [February 7, 2023, 4:36pm UTC](https://discuss.elastic.co/t/three-node-cluster-failes-completely-when-one-node-shuts-down/322638/11 "2023-02-07T16:36:56Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.