Hello,
I build a 11 node Elastic cluster (3M, 5D, 3 Ingest nodes), using ansible playbook. Which is working perfectly fine with out SSL & TLS. But when i enable SSL & TLS, i'am getting the below error message in master node. when i browse the master node using 9200, i' am getting master not discovered error exception message. Can somebody help on this pls. Please be noted we are using CS signed certs.
Here is the config file we are using
bootstrap.memory_lock: true
cluster.initial_master_nodes:
- ***prdelkmstr01
- ***prdelkmstr02
- ***prdelkmstr03
cluster.name: ***-prd-cluster
discovery.seed_hosts:
- ***prdelkmstr01
- ***prdelkmstr02
- ***prdelkmstr03
http.port: 9200
network.host: _site_,_local_
node.attr.tag: master
node.data: false
node.ingest: false
node.master: true
xpack.security.http.ssl.verification_mode: certificate
node.name: ***elkmstr01
path.data: /data/elasticsearch-prod
path.logs: /data/elasticsearch-logs
action.auto_create_index: true
xpack.security.enabled: true
xpack.security.transport.ssl.enabled: true
xpack.security.transport.ssl.verification_mode: "certificate"
xpack.security.transport.ssl.key: "/etc/elasticsearch/certs/elkprd.key"
xpack.security.transport.ssl.certificate: "/etc/elasticsearch/certs/elkprd.cer"
xpack.security.transport.ssl.certificate_authorities: "/etc/elasticsearch/certs/elkprd.crt"
xpack.security.http.ssl.enabled: true
xpack.security.http.ssl.key: "/etc/elasticsearch/certs/elkprd.key"
xpack.security.http.ssl.certificate: "/etc/elasticsearch/certs/elkprd.cer"
xpack.security.http.ssl.certificate_authorities: "/etc/elasticsearch/certs/elkprd.crt"
///////////////////////////////////////////////////////////////////////////////////////////////////////////
ERROR MESSAGE
///////////////////////////////////////////////////////////////////////////////////////////////////////////
[2022-02-11T08:17:03,725][WARN ][o.e.c.c.ClusterFormationFailureHelper] [********mstr01] master not discovered or elected yet, an election requires at least 2 nodes with ids from [40AiHIkDREWWP6lJWCzo5w, qbNJj26cTLiq1oUu28Wusg, bneZFy1sQR6QGF7NcfeTew], have discovered [{mo1prdelkmstr01}{qbNJj26cTLiq1oUu28Wusg}{P7MdpWjMS1KW3WyhhNPjhA}{10.115.113.21}{10.115.113.21:9300}{lmr}] which is not a quorum; discovery will continue using [10.115.113.24:9300, 10.115.113.25:9300] from hosts providers and [{mo1prdelkmstr01}{qbNJj26cTLiq1oUu28Wusg}{P7MdpWjMS1KW3WyhhNPjhA}{10.115.113.21}{10.115.113.21:9300}{lmr}] from last-known cluster state; node term 1, last-accepted version 60 in term 1
[2022-02-11T08:17:03,727][WARN ][o.e.t.TcpTransport ] [*******mstr01] exception caught on transport layer [Netty4TcpChannel{localAddress=/10.115.113.21:9300, remoteAddress=/10.115.113.27:58434, profile=default}], closing connection
io.netty.handler.codec.DecoderException: javax.net.ssl.SSLHandshakeException: Empty client certificate chain
at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:471) ~[netty-codec-4.1.49.Final.jar:4.1.49.Final]
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:276) ~[netty-codec-4.1.49.Final.jar:4.1.49.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:615) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:578) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) [netty-common-4.1.49.Final.jar:4.1.49.Final]
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) [netty-common-4.1.49.Final.jar:4.1.49.Final]
at java.lang.Thread.run(Thread.java:831) [?:?]
Caused by: javax.net.ssl.SSLHandshakeException: Empty client certificate chain
at sun.security.ssl.Alert.createSSLException(Alert.java:131) ~[?:?]
at sun.security.ssl.Alert.createSSLException(Alert.java:117) ~[?:?]
at sun.security.ssl.TransportContext.fatal(TransportContext.java:356) ~[?:?]
at sun.security.ssl.TransportContext.fatal(TransportContext.java:312) ~[?:?]
at sun.security.ssl.TransportContext.fatal(TransportContext.java:303) ~[?:?]
at sun.security.ssl.CertificateMessage$T13CertificateConsumer.onConsumeCertificate(CertificateMessage.java:1194) ~[?:?]
at sun.security.ssl.CertificateMessage$T13CertificateConsumer.consume(CertificateMessage.java:1181) ~[?:?]
at sun.security.ssl.SSLHandshake.consume(SSLHandshake.java:396) ~[?:?]
at sun.security.ssl.HandshakeContext.dispatch(HandshakeContext.java:480) ~[?:?]
at sun.security.ssl.SSLEngineImpl$DelegatedTask$DelegatedAction.run(SSLEngineImpl.java:1267) ~[?:?]
at sun.security.ssl.SSLEngineImpl$DelegatedTask$DelegatedAction.run(SSLEngineImpl.java:1254) ~[?:?]
at java.security.AccessController.doPrivileged(AccessController.java:691) ~[?:?]
at sun.security.ssl.SSLEngineImpl$DelegatedTask.run(SSLEngineImpl.java:1199) ~[?:?]
at io.netty.handler.ssl.SslHandler.runAllDelegatedTasks(SslHandler.java:1542) ~[netty-handler-4.1.49.Final.jar:4.1.49.Final]
at io.netty.handler.ssl.SslHandler.runDelegatedTasks(SslHandler.java:1556) ~[netty-handler-4.1.49.Final.jar:4.1.49.Final]
at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1440) ~[netty-handler-4.1.49.Final.jar:4.1.49.Final]
at io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1267) ~[netty-handler-4.1.49.Final.jar:4.1.49.Final]
at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1314) ~[netty-handler-4.1.49.Final.jar:4.1.49.Final]
at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:501) ~[netty-codec-4.1.49.Final.jar:4.1.49.Final]
at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:440) ~[netty-codec-4.1.49.Final.jar:4.1.49.Final]
... 16 more