Hi All,
We are new to using Elasticsearch and we use Elasticsearch as audittrail for Informatica PIM. We were able to successfully create Elasticsearch nodes and enable SSL in DEV and SIT which have standalone nodes but while setting up ssl in a cluster for PERF environment we are seeing handshake errors as below:
Error log:
[2022-05-27T03:46:51,431][INFO ][o.e.p.PluginsService ] [node-2] loaded module [aggs-matrix-stats]
[2022-05-27T03:46:51,431][INFO ][o.e.p.PluginsService ] [node-2] loaded module [analysis-common]
[2022-05-27T03:46:51,432][INFO ][o.e.p.PluginsService ] [node-2] loaded module [constant-keyword]
[2022-05-27T03:46:51,432][INFO ][o.e.p.PluginsService ] [node-2] loaded module [flattened]
[2022-05-27T03:46:51,432][INFO ][o.e.p.PluginsService ] [node-2] loaded module [frozen-indices]
[2022-05-27T03:46:51,432][INFO ][o.e.p.PluginsService ] [node-2] loaded module [ingest-common]
[2022-05-27T03:46:51,432][INFO ][o.e.p.PluginsService ] [node-2] loaded module [ingest-geoip]
[2022-05-27T03:46:51,432][INFO ][o.e.p.PluginsService ] [node-2] loaded module [ingest-user-agent]
[2022-05-27T03:46:51,432][INFO ][o.e.p.PluginsService ] [node-2] loaded module [lang-expression]
[2022-05-27T03:46:51,432][INFO ][o.e.p.PluginsService ] [node-2] loaded module [lang-mustache]
[2022-05-27T03:46:51,433][INFO ][o.e.p.PluginsService ] [node-2] loaded module [lang-painless]
[2022-05-27T03:46:51,433][INFO ][o.e.p.PluginsService ] [node-2] loaded module [mapper-extras]
[2022-05-27T03:46:51,433][INFO ][o.e.p.PluginsService ] [node-2] loaded module [parent-join]
[2022-05-27T03:46:51,433][INFO ][o.e.p.PluginsService ] [node-2] loaded module [percolator]
[2022-05-27T03:46:51,433][INFO ][o.e.p.PluginsService ] [node-2] loaded module [rank-eval]
[2022-05-27T03:46:51,433][INFO ][o.e.p.PluginsService ] [node-2] loaded module [reindex]
[2022-05-27T03:46:51,433][INFO ][o.e.p.PluginsService ] [node-2] loaded module [repository-url]
[2022-05-27T03:46:51,434][INFO ][o.e.p.PluginsService ] [node-2] loaded module [search-business-rules]
[2022-05-27T03:46:51,434][INFO ][o.e.p.PluginsService ] [node-2] loaded module [spatial]
[2022-05-27T03:46:51,434][INFO ][o.e.p.PluginsService ] [node-2] loaded module [tasks]
[2022-05-27T03:46:51,434][INFO ][o.e.p.PluginsService ] [node-2] loaded module [transform]
[2022-05-27T03:46:51,434][INFO ][o.e.p.PluginsService ] [node-2] loaded module [transport-netty4]
[2022-05-27T03:46:51,434][INFO ][o.e.p.PluginsService ] [node-2] loaded module [vectors]
[2022-05-27T03:46:51,434][INFO ][o.e.p.PluginsService ] [node-2] loaded module [x-pack-analytics]
[2022-05-27T03:46:51,434][INFO ][o.e.p.PluginsService ] [node-2] loaded module [x-pack-async-search]
[2022-05-27T03:46:51,435][INFO ][o.e.p.PluginsService ] [node-2] loaded module [x-pack-autoscaling]
[2022-05-27T03:46:51,435][INFO ][o.e.p.PluginsService ] [node-2] loaded module [x-pack-ccr]
[2022-05-27T03:46:51,435][INFO ][o.e.p.PluginsService ] [node-2] loaded module [x-pack-core]
[2022-05-27T03:46:51,435][INFO ][o.e.p.PluginsService ] [node-2] loaded module [x-pack-deprecation]
[2022-05-27T03:46:51,435][INFO ][o.e.p.PluginsService ] [node-2] loaded module [x-pack-enrich]
[2022-05-27T03:46:51,435][INFO ][o.e.p.PluginsService ] [node-2] loaded module [x-pack-eql]
[2022-05-27T03:46:51,435][INFO ][o.e.p.PluginsService ] [node-2] loaded module [x-pack-graph]
[2022-05-27T03:46:51,435][INFO ][o.e.p.PluginsService ] [node-2] loaded module [x-pack-identity-provider]
[2022-05-27T03:46:51,436][INFO ][o.e.p.PluginsService ] [node-2] loaded module [x-pack-ilm]
[2022-05-27T03:46:51,436][INFO ][o.e.p.PluginsService ] [node-2] loaded module [x-pack-logstash]
[2022-05-27T03:46:51,436][INFO ][o.e.p.PluginsService ] [node-2] loaded module [x-pack-ml]
[2022-05-27T03:46:51,436][INFO ][o.e.p.PluginsService ] [node-2] loaded module [x-pack-monitoring]
[2022-05-27T03:46:51,436][INFO ][o.e.p.PluginsService ] [node-2] loaded module [x-pack-ql]
[2022-05-27T03:46:51,436][INFO ][o.e.p.PluginsService ] [node-2] loaded module [x-pack-rollup]
[2022-05-27T03:46:51,436][INFO ][o.e.p.PluginsService ] [node-2] loaded module [x-pack-security]
[2022-05-27T03:46:51,436][INFO ][o.e.p.PluginsService ] [node-2] loaded module [x-pack-sql]
[2022-05-27T03:46:51,436][INFO ][o.e.p.PluginsService ] [node-2] loaded module [x-pack-voting-only-node]
[2022-05-27T03:46:51,437][INFO ][o.e.p.PluginsService ] [node-2] loaded module [x-pack-watcher]
[2022-05-27T03:46:51,437][INFO ][o.e.p.PluginsService ] [node-2] no plugins loaded
[2022-05-27T03:46:54,319][INFO ][o.e.x.s.a.s.FileRolesStore] [node-2] parsed [0] roles from file [/u01/app/elasticsearch/elasticsearch-7.7.0/config/roles.yml]
[2022-05-27T03:46:54,670][INFO ][o.e.x.m.p.l.CppLogMessageHandler] [node-2] [controller/3530011] [Main.cc@110] controller (64 bit): Version 7.7.0 (Build a8939d3da43f33) Copyright (c) 2020 Elasticsearch BV
[2022-05-27T03:46:55,239][INFO ][o.e.d.DiscoveryModule ] [node-2] using discovery type [zen] and seed hosts providers [settings]
[2022-05-27T03:46:55,916][INFO ][o.e.n.Node ] [node-2] initialized
[2022-05-27T03:46:55,916][INFO ][o.e.n.Node ] [node-2] starting ...
[2022-05-27T03:46:56,113][INFO ][o.e.t.TransportService ] [node-2] publish_address {infnlpimperf**.****.dell.com/10.***.***.***:9300}, bound_addresses {10.***.***.***:9300}
[2022-05-27T03:46:56,323][INFO ][o.e.b.BootstrapChecks ] [node-2] bound or publishing to a non-loopback address, enforcing bootstrap checks
[2022-05-27T03:46:56,326][INFO ][o.e.c.c.Coordinator ] [node-2] cluster UUID [bf3jVM9WT_KoA-B6bcfQTg]
[2022-05-27T03:46:56,553][WARN ][o.e.t.TcpTransport ] [node-2] exception caught on transport layer [Netty4TcpChannel{localAddress=0.0.0.0/0.0.0.0:11122, remoteAddress=infnlpimperf**.****.dell.com/10.***.***.***:9300}], closing connection
io.netty.handler.codec.DecoderException: javax.net.ssl.SSLHandshakeException: Received fatal alert: bad_certificate
at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:468) ~[netty-codec-4.1.45.Final.jar:4.1.45.Final]
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:276) ~[netty-codec-4.1.45.Final.jar:4.1.45.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377) [netty-transport-4.1.45.Final.jar:4.1.45.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363) [netty-transport-4.1.45.Final.jar:4.1.45.Final]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355) [netty-transport-4.1.45.Final.jar:4.1.45.Final]
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) [netty-transport-4.1.45.Final.jar:4.1.45.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377) [netty-transport-4.1.45.Final.jar:4.1.45.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363) [netty-transport-4.1.45.Final.jar:4.1.45.Final]
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) [netty-transport-4.1.45.Final.jar:4.1.45.Final]
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) [netty-transport-4.1.45.Final.jar:4.1.45.Final]
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714) [netty-transport-4.1.45.Final.jar:4.1.45.Final]
at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:615) [netty-transport-4.1.45.Final.jar:4.1.45.Final]
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:578) [netty-transport-4.1.45.Final.jar:4.1.45.Final]
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) [netty-transport-4.1.45.Final.jar:4.1.45.Final]
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) [netty-common-4.1.45.Final.jar:4.1.45.Final]
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) [netty-common-4.1.45.Final.jar:4.1.45.Final]
at java.lang.Thread.run(Thread.java:750) [?:1.8.0_321]
Caused by: javax.net.ssl.SSLHandshakeException: Received fatal alert: bad_certificate
at sun.security.ssl.Alert.createSSLException(Alert.java:131) ~[?:?]
at sun.security.ssl.Alert.createSSLException(Alert.java:117) ~[?:?]
at sun.security.ssl.TransportContext.fatal(TransportContext.java:356) ~[?:?]
at sun.security.ssl.Alert$AlertConsumer.consume(Alert.java:293) ~[?:?]
at sun.security.ssl.TransportContext.dispatch(TransportContext.java:202) ~[?:?]
at sun.security.ssl.SSLTransport.decode(SSLTransport.java:155) ~[?:?]
at sun.security.ssl.SSLEngineImpl.decode(SSLEngineImpl.java:597) ~[?:?]
at sun.security.ssl.SSLEngineImpl.readRecord(SSLEngineImpl.java:552) ~[?:?]
at sun.security.ssl.SSLEngineImpl.unwrap(SSLEngineImpl.java:418) ~[?:?]
at sun.security.ssl.SSLEngineImpl.unwrap(SSLEngineImpl.java:397) ~[?:?]
at javax.net.ssl.SSLEngine.unwrap(SSLEngine.java:626) ~[?:1.8.0_321]
at io.netty.handler.ssl.SslHandler$SslEngineType$3.unwrap(SslHandler.java:281) ~[netty-handler-4.1.45.Final.jar:4.1.45.Final]
at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1324) ~[netty-handler-4.1.45.Final.jar:4.1.45.Final]
at io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1219) ~[netty-handler-4.1.45.Final.jar:4.1.45.Final]
at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1266) ~[netty-handler-4.1.45.Final.jar:4.1.45.Final]
at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:498) ~[netty-codec-4.1.45.Final.jar:4.1.45.Final]
at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:437) ~[netty-codec-4.1.45.Final.jar:4.1.45.Final]
... 16 more
[2022-05-27T03:46:57,374][WARN ][o.e.t.TcpTransport ] [node-2] exception caught on transport layer [Netty4TcpChannel{localAddress=0.0.0.0/0.0.0.0:11124, remoteAddress=infnlpimperf**.****.dell.com/10.***.***.***:9300}], closing connection
io.netty.handler.codec.DecoderException: javax.net.ssl.SSLHandshakeException: Received fatal alert: bad_certificate
at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:468) ~[netty-codec-4.1.45.Final.jar:4.1.45.Final]
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:276) ~[netty-codec-4.1.45.Final.jar:4.1.45.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377) [netty-transport-4.1.45.Final.jar:4.1.45.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363) [netty-transport-4.1.45.Final.jar:4.1.45.Final]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355) [netty-transport-4.1.45.Final.jar:4.1.45.Final]
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) [netty-transport-4.1.45.Final.jar:4.1.45.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377) [netty-transport-4.1.45.Final.jar:4.1.45.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363) [netty-transport-4.1.45.Final.jar:4.1.45.Final]
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) [netty-transport-4.1.45.Final.jar:4.1.45.Final]
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) [netty-transport-4.1.45.Final.jar:4.1.45.Final]
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714) [netty-transport-4.1.45.Final.jar:4.1.45.Final]
at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:615) [netty-transport-4.1.45.Final.jar:4.1.45.Final]
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:578) [netty-transport-4.1.45.Final.jar:4.1.45.Final]
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) [netty-transport-4.1.45.Final.jar:4.1.45.Final]
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) [netty-common-4.1.45.Final.jar:4.1.45.Final]
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) [netty-common-4.1.45.Final.jar:4.1.45.Final]
at java.lang.Thread.run(Thread.java:750) [?:1.8.0_321]
Caused by: javax.net.ssl.SSLHandshakeException: Received fatal alert: bad_certificate
Below is the Elasticsearch.yml file configuration:
<
$ vi Elasticsearch.yml
Add custom attributes to the node:
#node.attr.rack: r1
----------------------------------- Paths ------------------------------------
Path to directory where to store the data (separate multiple locations by comma):
path.data: /u01/app/data
Path to log files:
path.logs: /u01/app/logs
----------------------------------- Memory -----------------------------------
Lock the memory on startup:
#bootstrap.memory_lock: true
Make sure that the heap size is set to about half the memory available
on the system and that the owner of the process is allowed to use this
limit.
Elasticsearch performs poorly when the system is swapping the memory.
---------------------------------- Network -----------------------------------
Set the bind address to a specific IP (IPv4 or IPv6):
network.host: infnlpimperf**.****.dell.com
Set a custom port for HTTP:
http.port: 9200
For more information, consult the network module documentation.
--------------------------------- Discovery ----------------------------------
Pass an initial list of hosts to perform discovery when this node is started:
The default list of hosts is ["127.0.0.1", "[::1]"]
discovery.seed_hosts: ["infnlpimperf**.****.dell.com"]
Bootstrap the cluster using an initial set of master-eligible nodes:
cluster.initial_master_nodes: ["node-1"]
For more information, consult the discovery and cluster formation module documentation.
#----------------------------------- SSL --------------------------------------
xpack.security.enabled: true
xpack.security.transport.ssl.enabled: true
xpack.security.transport.ssl.verification_mode: certificate
xpack.security.transport.ssl.key: /u01/app/Elasticsearch/Elasticsearch-7.7.0/config/keystore_decrypt.key
xpack.security.transport.ssl.certificate: /u01/app/Elasticsearch/Elasticsearch-7.7.0/config/keystore.crt
xpack.security.transport.ssl.truststore.path: /u01/app/Elasticsearch/Elasticsearch-7.7.0/config/keystore.p12
xpack.security.transport.ssl.truststore.password: **************
xpack.security.http.ssl.enabled: true
xpack.security.http.ssl.key: /u01/app/Elasticsearch/Elasticsearch-7.7.0/config/keystore_decrypt.key
xpack.security.http.ssl.certificate: /u01/app/Elasticsearch/Elasticsearch-7.7.0/config/keystore.crt
xpack.security.http.ssl.truststore.path: /u01/app/Elasticsearch/Elasticsearch-7.7.0/config/keystore.p12
xpack.security.http.ssl.truststore.password: ***************
xpack.security.http.ssl.client_authentication: optional
---------------------------------- Gateway -----------------------------------
Block initial recovery after a full cluster restart until N nodes are started:
#gateway.recover_after_nodes: 2
For more information, consult the gateway module documentation.
---------------------------------- Various -----------------------------------
Require explicit names when deleting indices:
#action.destructive_requires_name: true
/>
We have checked the certs and it has names of all the servers which are part of cluster. We have imported the certificates in cacerts files on the server.
Can some one help me understand what I could be doing wrong?