Failed to setup SSL Elasticsearch

Elastic Stack Elasticsearch
1

Hi All,

I have tried to implement SSL for ELK which has 3 nodes in the cluster. So SAN certificate has been generated with all 3 node’s hostname and FQDN and exported as .pfx (PKCS12) format to configure same in elasticsearch.yml, please find below config detail. Both keystore and truststore password are stored in default elastic keystore.

Post changes, Elasticsearch windows services are started in all 3 nodes in the cluster but the nodes were failed to connect and getting unexpected response code [503] as mentioned below.

Unexpected response code [503] from calling GET https://*:9200/_cluster/health?pretty

Cause: master_not_discovered_exception

Due to above error I couldn’t able to create default account credential, getting Failed to determine the health of the cluster running at https://)))): 9200))):%209200)

Elasticsearch.yml file keystore and truststore password stored in default keystore

xpack.security.enabled: true
xpack.http.ssl.verification_mode: certificate

xpack.security.http.ssl:
  enabled: true
  verification_mode: certificate
  keystore.path: 'certs\elasticsearch.pfx'
  truststore.path: 'certs\elasticsearch.pfx'

xpack.security.transport.ssl:
  enabled: true
  verification_mode: certificate
  keystore.path: 'certs\elasticsearch pfx'
  truststore.path: 'certs\elasticsearch.pfx'

Could someone help me to get ride of this issues to enable security settings?

Thanks in advance!

Regards,
Sathiya

2

Please share server logs for troubleshooting.

3

Thanks for checking, FYI, please find below logs;

[2022-08-17T00:05:51,031][INFO ][o.e.n.Node               ] [ELKMaster.domin.local] stopping ...
[2022-08-17T00:05:51,047][INFO ][o.e.x.w.WatcherService   ] [ELKMaster.domin.local] stopping watch service, reason [shutdown initiated]
[2022-08-17T00:05:51,047][INFO ][o.e.x.m.p.l.CppLogMessageHandler] [ELKMaster.domin.local] [controller/6836] [Main.cc@154] ML controller exiting
[2022-08-17T00:05:51,052][INFO ][o.e.x.m.p.NativeController] [ELKMaster.domin.local] Native controller process has stopped - no new native processes can be started
[2022-08-17T00:05:51,052][INFO ][o.e.x.w.WatcherLifeCycleService] [ELKMaster.domin.local] watcher has stopped and shutdown
[2022-08-17T00:05:51,278][INFO ][o.e.n.Node               ] [ELKMaster.domin.local] stopped
[2022-08-17T00:05:51,278][INFO ][o.e.n.Node               ] [ELKMaster.domin.local] closing ...
[2022-08-17T00:05:51,294][INFO ][o.e.n.Node               ] [ELKMaster.domin.local] closed
[2022-08-17T00:05:58,971][INFO ][o.e.n.Node               ] [ELKMaster.domin.local] version[7.10.0], pid[4924], build[unknown/unknown/51e9d6f22758d0374a0f3f5c6e8f3a7997850f96/2020-11-09T21:30:33.964949Z], OS[Windows Server 2019/10.0/amd64], JVM[AdoptOpenJDK/OpenJDK 64-Bit Server VM/15.0.1/15.0.1+9]
[2022-08-17T00:05:58,971][INFO ][o.e.n.Node               ] [ELKMaster.domin.local] JVM home [E:\Apps\Elastic\Elasticsearch\7.10.0\jdk]
[2022-08-17T00:05:58,971][INFO ][o.e.n.Node               ] [ELKMaster.domin.local] JVM arguments [-Des.networkaddress.cache.ttl=60, -Des.networkaddress.cache.negative.ttl=10, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -XX:-OmitStackTraceInFastThrow, -XX:+ShowCodeDetailsInExceptionMessages, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dio.netty.allocator.numDirectArenas=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Djava.locale.providers=SPI,COMPAT, -XX:+UseG1GC, -XX:G1ReservePercent=25, -XX:InitiatingHeapOccupancyPercent=30, -Djava.io.tmpdir=C:\Windows\TEMP\elasticsearch, -XX:+HeapDumpOnOutOfMemoryError, -XX:HeapDumpPath=data, -XX:ErrorFile=logs/hs_err_pid%p.log, -Xlog:gc*,gc+age=trace,safepoint:file=logs/gc.log:utctime,pid,tags:filecount=32,filesize=64m, -Xmx4096m, -Xms4096m, -XX:MaxDirectMemorySize=2147483648, -Delasticsearch, -Des.path.home=E:\Apps\Elastic\Elasticsearch\7.10.0, -Des.path.conf=E:\Apps\Elastic\Elasticsearch\Config, -agentpath:C:\Program Files\Palo Alto Networks\Traps\cyjagent.dll]
[2022-08-17T00:06:02,424][INFO ][o.e.p.PluginsService     ] [ELKMaster.domin.local] loaded module [aggs-matrix-stats]
[2022-08-17T00:06:02,424][INFO ][o.e.p.PluginsService     ] [ELKMaster.domin.local] loaded module [analysis-common]
[2022-08-17T00:06:02,424][INFO ][o.e.p.PluginsService     ] [ELKMaster.domin.local] loaded module [constant-keyword]
[2022-08-17T00:06:02,424][INFO ][o.e.p.PluginsService     ] [ELKMaster.domin.local] loaded module [flattened]
[2022-08-17T00:06:02,424][INFO ][o.e.p.PluginsService     ] [ELKMaster.domin.local] loaded module [frozen-indices]
[2022-08-17T00:06:02,424][INFO ][o.e.p.PluginsService     ] [ELKMaster.domin.local] loaded module [ingest-common]
[2022-08-17T00:06:02,424][INFO ][o.e.p.PluginsService     ] [ELKMaster.domin.local] loaded module [ingest-geoip]
[2022-08-17T00:06:02,424][INFO ][o.e.p.PluginsService     ] [ELKMaster.domin.local] loaded module [ingest-user-agent]
[2022-08-17T00:06:02,424][INFO ][o.e.p.PluginsService     ] [ELKMaster.domin.local] loaded module [kibana]
[2022-08-17T00:06:02,439][INFO ][o.e.p.PluginsService     ] [ELKMaster.domin.local] loaded module [lang-expression]
[2022-08-17T00:06:02,439][INFO ][o.e.p.PluginsService     ] [ELKMaster.domin.local] loaded module [lang-mustache]
[2022-08-17T00:06:02,439][INFO ][o.e.p.PluginsService     ] [ELKMaster.domin.local] loaded module [lang-painless]
[2022-08-17T00:06:02,439][INFO ][o.e.p.PluginsService     ] [ELKMaster.domin.local] loaded module [mapper-extras]
[2022-08-17T00:06:02,439][INFO ][o.e.p.PluginsService     ] [ELKMaster.domin.local] loaded module [mapper-version]
[2022-08-17T00:06:02,439][INFO ][o.e.p.PluginsService     ] [ELKMaster.domin.local] loaded module [parent-join]
[2022-08-17T00:06:02,439][INFO ][o.e.p.PluginsService     ] [ELKMaster.domin.local] loaded module [percolator]
[2022-08-17T00:06:02,439][INFO ][o.e.p.PluginsService     ] [ELKMaster.domin.local] loaded module [rank-eval]
[2022-08-17T00:06:02,439][INFO ][o.e.p.PluginsService     ] [ELKMaster.domin.local] loaded module [reindex]
[2022-08-17T00:06:02,439][INFO ][o.e.p.PluginsService     ] [ELKMaster.domin.local] loaded module [repositories-metering-api]
[2022-08-17T00:06:02,439][INFO ][o.e.p.PluginsService     ] [ELKMaster.domin.local] loaded module [repository-url]
[2022-08-17T00:06:02,439][INFO ][o.e.p.PluginsService     ] [ELKMaster.domin.local] loaded module [search-business-rules]
[2022-08-17T00:06:02,439][INFO ][o.e.p.PluginsService     ] [ELKMaster.domin.local] loaded module [searchable-snapshots]
[2022-08-17T00:06:02,439][INFO ][o.e.p.PluginsService     ] [ELKMaster.domin.local] loaded module [spatial]
[2022-08-17T00:06:02,439][INFO ][o.e.p.PluginsService     ] [ELKMaster.domin.local] loaded module [transform]
[2022-08-17T00:06:02,439][INFO ][o.e.p.PluginsService     ] [ELKMaster.domin.local] loaded module [transport-netty4]
[2022-08-17T00:06:02,439][INFO ][o.e.p.PluginsService     ] [ELKMaster.domin.local] loaded module [unsigned-long]
[2022-08-17T00:06:02,439][INFO ][o.e.p.PluginsService     ] [ELKMaster.domin.local] loaded module [vectors]
[2022-08-17T00:06:02,439][INFO ][o.e.p.PluginsService     ] [ELKMaster.domin.local] loaded module [wildcard]
[2022-08-17T00:06:02,439][INFO ][o.e.p.PluginsService     ] [ELKMaster.domin.local] loaded module [x-pack-analytics]
[2022-08-17T00:06:02,439][INFO ][o.e.p.PluginsService     ] [ELKMaster.domin.local] loaded module [x-pack-async]
[2022-08-17T00:06:02,439][INFO ][o.e.p.PluginsService     ] [ELKMaster.domin.local] loaded module [x-pack-async-search]
[2022-08-17T00:06:02,439][INFO ][o.e.p.PluginsService     ] [ELKMaster.domin.local] loaded module [x-pack-autoscaling]
[2022-08-17T00:06:02,439][INFO ][o.e.p.PluginsService     ] [ELKMaster.domin.local] loaded module [x-pack-ccr]
[2022-08-17T00:06:02,439][INFO ][o.e.p.PluginsService     ] [ELKMaster.domin.local] loaded module [x-pack-core]
[2022-08-17T00:06:02,439][INFO ][o.e.p.PluginsService     ] [ELKMaster.domin.local] loaded module [x-pack-data-streams]
[2022-08-17T00:06:02,439][INFO ][o.e.p.PluginsService     ] [ELKMaster.domin.local] loaded module [x-pack-deprecation]
[2022-08-17T00:06:02,439][INFO ][o.e.p.PluginsService     ] [ELKMaster.domin.local] loaded module [x-pack-enrich]
[2022-08-17T00:06:02,439][INFO ][o.e.p.PluginsService     ] [ELKMaster.domin.local] loaded module [x-pack-eql]
[2022-08-17T00:06:02,439][INFO ][o.e.p.PluginsService     ] [ELKMaster.domin.local] loaded module [x-pack-graph]
[2022-08-17T00:06:02,439][INFO ][o.e.p.PluginsService     ] [ELKMaster.domin.local] loaded module [x-pack-identity-provider]
[2022-08-17T00:06:02,439][INFO ][o.e.p.PluginsService     ] [ELKMaster.domin.local] loaded module [x-pack-ilm]
[2022-08-17T00:06:02,439][INFO ][o.e.p.PluginsService     ] [ELKMaster.domin.local] loaded module [x-pack-logstash]
[2022-08-17T00:06:02,439][INFO ][o.e.p.PluginsService     ] [ELKMaster.domin.local] loaded module [x-pack-ml]
[2022-08-17T00:06:02,439][INFO ][o.e.p.PluginsService     ] [ELKMaster.domin.local] loaded module [x-pack-monitoring]
[2022-08-17T00:06:02,439][INFO ][o.e.p.PluginsService     ] [ELKMaster.domin.local] loaded module [x-pack-ql]
[2022-08-17T00:06:02,439][INFO ][o.e.p.PluginsService     ] [ELKMaster.domin.local] loaded module [x-pack-rollup]
[2022-08-17T00:06:02,439][INFO ][o.e.p.PluginsService     ] [ELKMaster.domin.local] loaded module [x-pack-security]
[2022-08-17T00:06:02,455][INFO ][o.e.p.PluginsService     ] [ELKMaster.domin.local] loaded module [x-pack-sql]
[2022-08-17T00:06:02,455][INFO ][o.e.p.PluginsService     ] [ELKMaster.domin.local] loaded module [x-pack-stack]
[2022-08-17T00:06:02,455][INFO ][o.e.p.PluginsService     ] [ELKMaster.domin.local] loaded module [x-pack-voting-only-node]
[2022-08-17T00:06:02,455][INFO ][o.e.p.PluginsService     ] [ELKMaster.domin.local] loaded module [x-pack-watcher]
[2022-08-17T00:06:02,455][INFO ][o.e.p.PluginsService     ] [ELKMaster.domin.local] no plugins loaded
[2022-08-17T00:06:02,939][INFO ][o.e.e.NodeEnvironment    ] [ELKMaster.domin.local] using [1] data paths, mounts [[(E:)]], net usable_space [464.6gb], net total_space [511.9gb], types [NTFS]
[2022-08-17T00:06:02,939][INFO ][o.e.e.NodeEnvironment    ] [ELKMaster.domin.local] heap size [4gb], compressed ordinary object pointers [true]
[2022-08-17T00:06:04,064][INFO ][o.e.n.Node               ] [ELKMaster.domin.local] node name [ELKMaster.domin.local], node ID [1m3oWjdmQ_CkVu9cbYrabg], cluster name [RPA_Orchestrator_Prod], roles [transform, master, remote_cluster_client, data, ml, data_content, data_hot, data_warm, data_cold, ingest]
[2022-08-17T00:06:08,660][INFO ][o.e.x.m.p.l.CppLogMessageHandler] [ELKMaster.domin.local] [controller/8808] [Main.cc@114] controller (64 bit): Version 7.10.0 (Build ac991e2e31f99d) Copyright (c) 2020 Elasticsearch BV
[2022-08-17T00:06:09,207][INFO ][o.e.x.s.a.s.FileRolesStore] [ELKMaster.domin.local] parsed [0] roles from file [E:\Apps\Elastic\Elasticsearch\Config\roles.yml]
[2022-08-17T00:06:10,484][INFO ][o.e.t.NettyAllocator     ] [ELKMaster.domin.local] creating NettyAllocator with the following configs: [name=elasticsearch_configured, chunk_size=512kb, suggested_max_allocation_size=512kb, factors={es.unsafe.use_netty_default_chunk_and_page_size=false, g1gc_enabled=true, g1gc_region_size=2mb}]
[2022-08-17T00:06:10,578][INFO ][o.e.d.DiscoveryModule    ] [ELKMaster.domin.local] using discovery type [zen] and seed hosts providers [settings]
[2022-08-17T00:06:11,173][WARN ][o.e.g.DanglingIndicesState] [ELKMaster.domin.local] gateway.auto_import_dangling_indices is disabled, dangling indices will not be automatically detected or imported and must be managed manually
[2022-08-17T00:06:11,689][INFO ][o.e.n.Node               ] [ELKMaster.domin.local] initialized
[2022-08-17T00:06:11,689][INFO ][o.e.n.Node               ] [ELKMaster.domin.local] starting ...
[2022-08-17T00:06:11,879][INFO ][o.e.t.TransportService   ] [ELKMaster.domin.local] publish_address {ELKMaster/xx.xx.xx.174:9300}, bound_addresses {[::1]:9300}, {xx.xx.xx.174:9300}
[2022-08-17T00:06:12,357][WARN ][o.e.t.TcpTransport       ] [ELKMaster.domin.local] exception caught on transport layer [Netty4TcpChannel{localAddress=/xx.xx.xx.174:9300, remoteAddress=/xx.xx.xx.176:64346}], closing connection
io.netty.handler.codec.DecoderException: javax.net.ssl.SSLHandshakeException: Empty client certificate chain
	at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:471) ~[netty-codec-4.1.49.Final.jar:4.1.49.Final]
	at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:276) ~[netty-codec-4.1.49.Final.jar:4.1.49.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
	at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
	at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
	at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
	at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
	at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:615) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
	at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:578) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
	at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) [netty-common-4.1.49.Final.jar:4.1.49.Final]
	at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) [netty-common-4.1.49.Final.jar:4.1.49.Final]
	at java.lang.Thread.run(Thread.java:832) [?:?]
Caused by: javax.net.ssl.SSLHandshakeException: Empty client certificate chain
	at sun.security.ssl.Alert.createSSLException(Alert.java:131) ~[?:?]
	at sun.security.ssl.Alert.createSSLException(Alert.java:117) ~[?:?]
	at sun.security.ssl.TransportContext.fatal(TransportContext.java:356) ~[?:?]
	at sun.security.ssl.TransportContext.fatal(TransportContext.java:312) ~[?:?]
	at sun.security.ssl.TransportContext.fatal(TransportContext.java:303) ~[?:?]
	at sun.security.ssl.CertificateMessage$T13CertificateConsumer.onConsumeCertificate(CertificateMessage.java:1194) ~[?:?]
	at sun.security.ssl.CertificateMessage$T13CertificateConsumer.consume(CertificateMessage.java:1181) ~[?:?]
	at sun.security.ssl.SSLHandshake.consume(SSLHandshake.java:396) ~[?:?]
	at sun.security.ssl.HandshakeContext.dispatch(HandshakeContext.java:480) ~[?:?]
	at sun.security.ssl.SSLEngineImpl$DelegatedTask$DelegatedAction.run(SSLEngineImpl.java:1267) ~[?:?]
	at sun.security.ssl.SSLEngineImpl$DelegatedTask$DelegatedAction.run(SSLEngineImpl.java:1254) ~[?:?]
	at java.security.AccessController.doPrivileged(AccessController.java:691) ~[?:?]
	at sun.security.ssl.SSLEngineImpl$DelegatedTask.run(SSLEngineImpl.java:1199) ~[?:?]
	at io.netty.handler.ssl.SslHandler.runAllDelegatedTasks(SslHandler.java:1542) ~[netty-handler-4.1.49.Final.jar:4.1.49.Final]
	at io.netty.handler.ssl.SslHandler.runDelegatedTasks(SslHandler.java:1556) ~[netty-handler-4.1.49.Final.jar:4.1.49.Final]
	at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1440) ~[netty-handler-4.1.49.Final.jar:4.1.49.Final]
	at io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1267) ~[netty-handler-4.1.49.Final.jar:4.1.49.Final]
	at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1314) ~[netty-handler-4.1.49.Final.jar:4.1.49.Final]
	at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:501) ~[netty-codec-4.1.49.Final.jar:4.1.49.Final]
	at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:440) ~[netty-codec-4.1.49.Final.jar:4.1.49.Final]
	... 16 more
[2022-08-17T00:06:12,341][WARN ][o.e.t.TcpTransport       ] [ELKMaster.domin.local] exception caught on transport layer [Netty4TcpChannel{localAddress=/xx.xx.xx.174:9300, remoteAddress=/xx.xx.xx.175:64769}], closing connection
io.netty.handler.codec.DecoderException: javax.net.ssl.SSLHandshakeException: Empty client certificate chain
	at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:471) ~[netty-codec-4.1.49.Final.jar:4.1.49.Final]
	at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:276) ~[netty-codec-4.1.49.Final.jar:4.1.49.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
	at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
	at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
	at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
	at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
	at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:615) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
	at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:578) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
	at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) [netty-common-4.1.49.Final.jar:4.1.49.Final]
	at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) [netty-common-4.1.49.Final.jar:4.1.49.Final]
	at java.lang.Thread.run(Thread.java:832) [?:?]
Caused by: javax.net.ssl.SSLHandshakeException: Empty client certificate chain
	at sun.security.ssl.Alert.createSSLException(Alert.java:131) ~[?:?]
	at sun.security.ssl.Alert.createSSLException(Alert.java:117) ~[?:?]
	at sun.security.ssl.TransportContext.fatal(TransportContext.java:356) ~[?:?]
	at sun.security.ssl.TransportContext.fatal(TransportContext.java:312) ~[?:?]
	at sun.security.ssl.TransportContext.fatal(TransportContext.java:303) ~[?:?]
	at sun.security.ssl.CertificateMessage$T13CertificateConsumer.onConsumeCertificate(CertificateMessage.java:1194) ~[?:?]
	at sun.security.ssl.CertificateMessage$T13CertificateConsumer.consume(CertificateMessage.java:1181) ~[?:?]
	at sun.security.ssl.SSLHandshake.consume(SSLHandshake.java:396) ~[?:?]
	at sun.security.ssl.HandshakeContext.dispatch(HandshakeContext.java:480) ~[?:?]
	at sun.security.ssl.SSLEngineImpl$DelegatedTask$DelegatedAction.run(SSLEngineImpl.java:1267) ~[?:?]
	at sun.security.ssl.SSLEngineImpl$DelegatedTask$DelegatedAction.run(SSLEngineImpl.java:1254) ~[?:?]
	at java.security.AccessController.doPrivileged(AccessController.java:691) ~[?:?]
	at sun.security.ssl.SSLEngineImpl$DelegatedTask.run(SSLEngineImpl.java:1199) ~[?:?]
	at io.netty.handler.ssl.SslHandler.runAllDelegatedTasks(SslHandler.java:1542) ~[netty-handler-4.1.49.Final.jar:4.1.49.Final]
	at io.netty.handler.ssl.SslHandler.runDelegatedTasks(SslHandler.java:1556) ~[netty-handler-4.1.49.Final.jar:4.1.49.Final]
	at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1440) ~[netty-handler-4.1.49.Final.jar:4.1.49.Final]
	at io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1267) ~[netty-handler-4.1.49.Final.jar:4.1.49.Final]
	at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1314) ~[netty-handler-4.1.49.Final.jar:4.1.49.Final]
	at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:501) ~[netty-codec-4.1.49.Final.jar:4.1.49.Final]
	at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:440) ~[netty-codec-4.1.49.Final.jar:4.1.49.Final]
	... 16 more
[2022-08-17T00:06:12,536][INFO ][o.e.b.BootstrapChecks    ] [ELKMaster.domin.local] bound or publishing to a non-loopback address, enforcing bootstrap checks
[2022-08-17T00:06:12,536][INFO ][o.e.c.c.Coordinator      ] [ELKMaster.domin.local] cluster UUID [eRDj9AT-TWmmvtJWJcLLow]
[2022-08-17T00:06:12,641][WARN ][o.e.t.TcpTransport       ] [ELKMaster.domin.local] exception caught on transport layer [Netty4TcpChannel{localAddress=/xx.xx.xx.174:55920, remoteAddress=ELKNode02/xx.xx.xx.176:9300}], closing connection
io.netty.handler.codec.DecoderException: javax.net.ssl.SSLHandshakeException: Received fatal alert: bad_certificate
	at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:471) ~[netty-codec-4.1.49.Final.jar:4.1.49.Final]
	at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:276) ~[netty-codec-4.1.49.Final.jar:4.1.49.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
	at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
	at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
	at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
	at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
	at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:615) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
	at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:578) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
	at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) [netty-common-4.1.49.Final.jar:4.1.49.Final]
	at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) [netty-common-4.1.49.Final.jar:4.1.49.Final]
	at java.lang.Thread.run(Thread.java:832) [?:?]
Caused by: javax.net.ssl.SSLHandshakeException: Received fatal alert: bad_certificate
	at sun.security.ssl.Alert.createSSLException(Alert.java:131) ~[?:?]
	at sun.security.ssl.Alert.createSSLException(Alert.java:117) ~[?:?]
	at sun.security.ssl.TransportContext.fatal(TransportContext.java:356) ~[?:?]
	at sun.security.ssl.Alert$AlertConsumer.consume(Alert.java:293) ~[?:?]
	at sun.security.ssl.TransportContext.dispatch(TransportContext.java:202) ~[?:?]
	at sun.security.ssl.SSLTransport.decode(SSLTransport.java:171) ~[?:?]
	at sun.security.ssl.SSLEngineImpl.decode(SSLEngineImpl.java:736) ~[?:?]
	at sun.security.ssl.SSLEngineImpl.readRecord(SSLEngineImpl.java:691) ~[?:?]
	at sun.security.ssl.SSLEngineImpl.unwrap(SSLEngineImpl.java:506) ~[?:?]
	at sun.security.ssl.SSLEngineImpl.unwrap(SSLEngineImpl.java:482) ~[?:?]
	at javax.net.ssl.SSLEngine.unwrap(SSLEngine.java:637) ~[?:?]
	at io.netty.handler.ssl.SslHandler$SslEngineType$3.unwrap(SslHandler.java:282) ~[netty-handler-4.1.49.Final.jar:4.1.49.Final]
	at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1372) ~[netty-handler-4.1.49.Final.jar:4.1.49.Final]
	at io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1267) ~[netty-handler-4.1.49.Final.jar:4.1.49.Final]
	at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1314) ~[netty-handler-4.1.49.Final.jar:4.1.49.Final]
	at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:501) ~[netty-codec-4.1.49.Final.jar:4.1.49.Final]
	at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:440) ~[netty-codec-4.1.49.Final.jar:4.1.49.Final]
	... 16 more
[2022-08-17T00:06:12,657][WARN ][o.e.t.TcpTransport       ] [ELKMaster.domin.local] exception caught on transport layer [Netty4TcpChannel{localAddress=/xx.xx.xx.174:55921, remoteAddress=ELKNode01/xx.xx.xx.175:9300}], closing connection
io.netty.handler.codec.DecoderException: javax.net.ssl.SSLHandshakeException: Received fatal alert: bad_certificate
	at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:471) ~[netty-codec-4.1.49.Final.jar:4
4 
[2022-08-17T00:06:12,341][WARN ][o.e.t.TcpTransport       ] [ELKMaster.domin.local] exception caught on transport layer [Netty4TcpChannel{localAddress=/xx.xx.xx.174:9300, remoteAddress=/xx.xx.xx.175:64769}], closing connection
io.netty.handler.codec.DecoderException: javax.net.ssl.SSLHandshakeException: Empty client certificate chain

The above error means the server (for which you shared the log) didn't get a certificate from the other side (ip xxx.175). Presumabally the other side didn't send one at all.

[2022-08-17T00:06:12,657][WARN ][o.e.t.TcpTransport       ] [ELKMaster.domin.local] exception caught on transport layer [Netty4TcpChannel{localAddress=/xx.xx.xx.174:55921, remoteAddress=enweuwpuip005/xx.xx.xx.175:9300}], closing connection
io.netty.handler.codec.DecoderException: javax.net.ssl.SSLHandshakeException: Received fatal alert: bad_certificate

This error means the other side (xx.175) didn't like the server the certificate.

You can:

  • Double check configuration on all 3 nodes.
  • Also check the content certs\ elasticsearch.pfx to see whether it the necessary certs and keys.
  • Check logs on the other side (xx.175) to see why it didn't like the server's certificate

Btw, you are using version 7.10 which is EOL. I suggest you update to latest 8.4 as soon as possible. It has security (including TLS) automatically enabled.

5

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.