Nodes disabled from Cluster when XPack SSL is enabled

Elastic Stack Elasticsearch
1

I am following the steps given in this

Node-1 elasticsearch.yml file

cluster.name: as-cluster
node.name: as-node1
node.master: true
node.data: true
network.host: 10.71.34.27
http.port: 9200
discovery.seed_hosts: ["10.71.34.27","10.71.33.105"]
cluster.initial_master_nodes: ["as-node1", "as-node2"]
xpack.security.enabled: true
xpack.security.transport.ssl.enabled: true
xpack.security.transport.ssl.verification_mode: certificate
xpack.security.transport.ssl.keystore.path: config/certs/elastic-certificates.p12
xpack.security.transport.ssl.truststore.path: config/certs/elastic-certificates.p12
xpack.security.http.ssl.enabled: true
xpack.security.http.ssl.keystore.path: config/certs/elastic-certificates.p12 
xpack.security.http.ssl.truststore.path: config/certs/elastic-certificates.p12

Almost same configuration for node-2

then when I try to fetch the cluster information after restarting, it returns

Empty Reply from server

and from the master node it says :

{"error":{"root_cause":[{"type":"master_not_discovered_exception","reason":null}]," type":"master_not_discovered_exception","reason":null},"status":503}

2

What do the Elasticsearch logs say?
Your cluster is not forming correctly, but it's hard for us to diagnose why without seeing the logs.

3

Hi @TimV the logs are as follow:

[2020-09-03T17:09:43,263][WARN ][o.e.x.c.s.t.n.SecurityNetty4Transport] [as-node3] client did not trust this server's certificate, closing connection Netty4TcpChannel{localAddress=/10.71.X.X:9300, remoteAddress=/10.71.Y.Y:42596}
[2020-09-03T17:09:44,258][WARN ][o.e.x.c.s.t.n.SecurityNetty4Transport] [as-node3] client did not trust this server's certificate, closing connection Netty4TcpChannel{localAddress=/10.71.X.X:9300, remoteAddress=/10.71.Y.Y:42606}
[2020-09-03T17:09:45,263][WARN ][o.e.x.c.s.t.n.SecurityNetty4Transport] [as-node3] client did not trust this server's certificate, closing connection Netty4TcpChannel{localAddress=/10.71.X.X:9300, remoteAddress=/10.71.Y.Y:42610}
[2020-09-03T17:09:46,185][WARN ][o.e.x.s.t.n.SecurityNetty4HttpServerTransport] [as-node3] received plaintext http traffic on an https channel, closing connection Netty4HttpChannel{localAddress=/10.71.X.X:9200, remoteAddress=/10.71.X.X:37722}
[2020-09-03T17:09:46,265][WARN ][o.e.x.c.s.t.n.SecurityNetty4Transport] [as-node3] client did not trust this server's certificate, closing connection Netty4TcpChannel{localAddress=/10.71.X.X:9300, remoteAddress=/10.71.Y.Y:42614}
[2020-09-03T17:09:47,272][WARN ][o.e.x.c.s.t.n.SecurityNetty4Transport] [as-node3] client did not trust this server's certificate, closing connection Netty4TcpChannel{localAddress=/10.71.X.X:9300, remoteAddress=/10.71.Y.Y:42618}
[2020-09-03T17:09:48,272][WARN ][o.e.x.c.s.t.n.SecurityNetty4Transport] [as-node3] client did not trust this server's certificate, closing connection Netty4TcpChannel{localAddress=/10.71.X.X:9300, remoteAddress=/10.71.Y.Y:42624}
[2020-09-03T17:09:49,260][WARN ][o.e.x.c.s.t.n.SecurityNetty4Transport] [as-node3] client did not trust this server's certificate, closing connection Netty4TcpChannel{localAddress=/10.71.X.X:9300, remoteAddress=/10.71.Y.Y:42628}
[2020-09-03T17:09:50,468][WARN ][o.e.x.c.s.t.n.SecurityNetty4Transport] [as-node3] client did not trust this server's certificate, closing connection Netty4TcpChannel{localAddress=/10.71.X.X:9300, remoteAddress=/10.71.Y.Y:42634}
[2020-09-03T17:09:51,198][WARN ][o.e.x.s.t.n.SecurityNetty4HttpServerTransport] [as-node3] received plaintext http traffic on an https channel, closing connection Netty4HttpChannel{localAddress=/10.71.X.X:9200, remoteAddress=/10.71.X.X:37734}
[2020-09-03T17:09:51,265][WARN ][o.e.x.c.s.t.n.SecurityNetty4Transport] [as-node3] client did not trust this server's certificate, closing connection Netty4TcpChannel{localAddress=/10.71.X.X:9300, remoteAddress=/10.71.Y.Y:42636}
[2020-09-03T17:09:51,533][INFO ][o.e.n.Node               ] [as-node3] stopping ...
[2020-09-03T17:09:51,541][INFO ][o.e.x.w.WatcherService   ] [as-node3] stopping watch service, reason [shutdown initiated]
[2020-09-03T17:09:51,541][INFO ][o.e.x.w.WatcherLifeCycleService] [as-node3] watcher has stopped and shutdown
[2020-09-03T17:09:51,567][INFO ][o.e.x.m.p.l.CppLogMessageHandler] [as-node3] [controller/10197] [Main.cc@150] Ml controller exiting
[2020-09-03T17:09:51,569][INFO ][o.e.x.m.p.NativeController] [as-node3] Native controller process has stopped - no new native processes can be started
[2020-09-03T17:09:51,620][INFO ][o.e.n.Node               ] [as-node3] stopped
[2020-09-03T17:09:51,620][INFO ][o.e.n.Node               ] [as-node3] closing ...
[2020-09-03T17:09:51,652][INFO ][o.e.n.Node               ] [as-node3] closed
[2020-09-03T17:09:54,137][INFO ][o.e.e.NodeEnvironment    ] [as-node3] using [1] data paths, mounts [[/ (/dev/sda2)]], net usable_space [38.1gb], net total_space [46.9gb], types [ext4]
[2020-09-03T17:09:54,140][INFO ][o.e.e.NodeEnvironment    ] [as-node3] heap size [1007.3mb], compressed ordinary object pointers [true]
[2020-09-03T17:09:54,146][INFO ][o.e.n.Node               ] [as-node3] node name [as-node3], node ID [YGkX-hT2RKO0W5HP9YKPbQ], cluster name [as-cluster]
[2020-09-03T17:09:54,146][INFO ][o.e.n.Node               ] [as-node3] version[7.4.2], pid[10258], build[default/rpm/2f90bbf7b93631e52bafb59b3b049cb44ec25e96/2019-10-28T20:40:44.881551Z], OS[Linux/4.12.14-120-default/amd64], JVM[AdoptOpenJDK/OpenJDK 64-Bit Server VM/13.0.1/13.0.1+9]
[2020-09-03T17:09:54,146][INFO ][o.e.n.Node               ] [as-node3] JVM home [/opt/novell/nam/elasticsearch/jdk]
[2020-09-03T17:09:54,147][INFO ][o.e.n.Node               ] [as-node3] JVM arguments [-Xms1g, -Xmx1g, -XX:+UseConcMarkSweepGC, -XX:CMSInitiatingOccupancyFraction=75, -XX:+UseCMSInitiatingOccupancyOnly, -Des.networkaddress.cache.ttl=60, -Des.networkaddress.cache.negative.ttl=10, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -XX:-OmitStackTraceInFastThrow, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dio.netty.allocator.numDirectArenas=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Djava.io.tmpdir=/tmp/elasticsearch-12067149196405027880, -XX:+HeapDumpOnOutOfMemoryError, -XX:HeapDumpPath=/var/lib/elasticsearch, -XX:ErrorFile=/var/opt/novell/nam/logs/elasticsearch/hs_err_pid%p.log, -Xlog:gc*,gc+age=trace,safepoint:file=/var/opt/novell/nam/logs/elasticsearch/gc.log:utctime,pid,tags:filecount=32,filesize=64m, -Djava.locale.providers=COMPAT, -Dio.netty.allocator.type=unpooled, -XX:MaxDirectMemorySize=536870912, -Des.path.home=/opt/novell/nam/elasticsearch, -Des.path.conf=/etc/elasticsearch, -Des.distribution.flavor=default, -Des.distribution.type=rpm, -Des.bundled_jdk=true]
[2020-09-03T17:09:56,684][INFO ][o.e.p.PluginsService     ] [as-node3] loaded module [aggs-matrix-stats]
[2020-09-03T17:09:56,685][INFO ][o.e.p.PluginsService     ] [as-node3] loaded module [analysis-common]
[2020-09-03T17:09:56,685][INFO ][o.e.p.PluginsService     ] [as-node3] loaded module [data-frame]
[2020-09-03T17:09:56,685][INFO ][o.e.p.PluginsService     ] [as-node3] loaded module [flattened]
[2020-09-03T17:09:56,685][INFO ][o.e.p.PluginsService     ] [as-node3] loaded module [frozen-indices]
[2020-09-03T17:09:56,686][INFO ][o.e.p.PluginsService     ] [as-node3] loaded module [ingest-common]
[2020-09-03T17:09:56,686][INFO ][o.e.p.PluginsService     ] [as-node3] loaded module [ingest-geoip]
[2020-09-03T17:09:56,686][INFO ][o.e.p.PluginsService     ] [as-node3] loaded module [ingest-user-agent]
[2020-09-03T17:09:56,686][INFO ][o.e.p.PluginsService     ] [as-node3] loaded module [lang-expression]
[2020-09-03T17:09:56,686][INFO ][o.e.p.PluginsService     ] [as-node3] loaded module [lang-mustache]
[2020-09-03T17:09:56,687][INFO ][o.e.p.PluginsService     ] [as-node3] loaded module [lang-painless]
[2020-09-03T17:09:56,687][INFO ][o.e.p.PluginsService     ] [as-node3] loaded module [mapper-extras]
[2020-09-03T17:09:56,687][INFO ][o.e.p.PluginsService     ] [as-node3] loaded module [parent-join]
[2020-09-03T17:09:56,687][INFO ][o.e.p.PluginsService     ] [as-node3] loaded module [percolator]
[2020-09-03T17:09:56,688][INFO ][o.e.p.PluginsService     ] [as-node3] loaded module [rank-eval]
[2020-09-03T17:09:56,688][INFO ][o.e.p.PluginsService     ] [as-node3] loaded module [reindex]
[2020-09-03T17:09:56,688][INFO ][o.e.p.PluginsService     ] [as-node3] loaded module [repository-url]
[2020-09-03T17:09:56,688][INFO ][o.e.p.PluginsService     ] [as-node3] loaded module [search-business-rules]
[2020-09-03T17:09:56,688][INFO ][o.e.p.PluginsService     ] [as-node3] loaded module [spatial]
[2020-09-03T17:09:56,689][INFO ][o.e.p.PluginsService     ] [as-node3] loaded module [systemd]
[2020-09-03T17:09:56,689][INFO ][o.e.p.PluginsService     ] [as-node3] loaded module [transport-netty4]
[2020-09-03T17:09:56,689][INFO ][o.e.p.PluginsService     ] [as-node3] loaded module [vectors]
[2020-09-03T17:09:56,689][INFO ][o.e.p.PluginsService     ] [as-node3] loaded module [x-pack-analytics]
[2020-09-03T17:09:56,689][INFO ][o.e.p.PluginsService     ] [as-node3] loaded module [x-pack-ccr]
[2020-09-03T17:09:56,690][INFO ][o.e.p.PluginsService     ] [as-node3] loaded module [x-pack-core]
[2020-09-03T17:09:56,690][INFO ][o.e.p.PluginsService     ] [as-node3] loaded module [x-pack-deprecation]
[2020-09-03T17:09:56,690][INFO ][o.e.p.PluginsService     ] [as-node3] loaded module [x-pack-graph]
[2020-09-03T17:09:56,690][INFO ][o.e.p.PluginsService     ] [as-node3] loaded module [x-pack-ilm]
[2020-09-03T17:09:56,690][INFO ][o.e.p.PluginsService     ] [as-node3] loaded module [x-pack-logstash]
[2020-09-03T17:09:56,691][INFO ][o.e.p.PluginsService     ] [as-node3] loaded module [x-pack-ml]
[2020-09-03T17:09:56,691][INFO ][o.e.p.PluginsService     ] [as-node3] loaded module [x-pack-monitoring]
[2020-09-03T17:09:56,691][INFO ][o.e.p.PluginsService     ] [as-node3] loaded module [x-pack-rollup]
[2020-09-03T17:09:56,691][INFO ][o.e.p.PluginsService     ] [as-node3] loaded module [x-pack-security]
[2020-09-03T17:09:56,691][INFO ][o.e.p.PluginsService     ] [as-node3] loaded module [x-pack-sql]
[2020-09-03T17:09:56,692][INFO ][o.e.p.PluginsService     ] [as-node3] loaded module [x-pack-voting-only-node]
[2020-09-03T17:09:56,692][INFO ][o.e.p.PluginsService     ] [as-node3] loaded module [x-pack-watcher]
[2020-09-03T17:09:56,692][INFO ][o.e.p.PluginsService     ] [as-node3] no plugins loaded
[2020-09-03T17:10:02,481][INFO ][o.e.x.s.a.s.FileRolesStore] [as-node3] parsed [0] roles from file [/etc/elasticsearch/roles.yml]
[2020-09-03T17:10:03,152][INFO ][o.e.x.m.p.l.CppLogMessageHandler] [as-node3] [controller/10350] [Main.cc@110] controller (64 bit): Version 7.4.2 (Build 473f61b8a5238b) Copyright (c) 2019 Elasticsearch BV
[2020-09-03T17:10:04,015][INFO ][o.e.d.DiscoveryModule    ] [as-node3] using discovery type [zen] and seed hosts providers [settings]
[2020-09-03T17:10:04,698][INFO ][o.e.n.Node               ] [as-node3] initialized
[2020-09-03T17:10:04,698][INFO ][o.e.n.Node               ] [as-node3] starting ...
[2020-09-03T17:10:04,802][INFO ][o.e.t.TransportService   ] [as-node3] publish_address {10.71.X.X:9300}, bound_addresses {10.71.X.X:9300}
[2020-09-03T17:10:04,810][INFO ][o.e.b.BootstrapChecks    ] [as-node3] bound or publishing to a non-loopback address, enforcing bootstrap checks
[2020-09-03T17:10:04,817][INFO ][o.e.c.c.Coordinator      ] [as-node3] cluster UUID [Yw9FqaPjSqe-SwV0eJQ63g]
[2020-09-03T17:10:05,109][WARN ][o.e.x.c.s.t.n.SecurityNetty4Transport] [as-node3] client did not trust this server's certificate, closing connection Netty4TcpChannel{localAddress=/10.71.X.X:44940, remoteAddress=/10.71.Y.Y:9300}
[2020-09-03T17:10:05,311][WARN ][o.e.x.c.s.t.n.SecurityNetty4Transport] [as-node3] client did not trust this server's certificate, closing connection
4

Now i have a three node cluster with configuration as follow:

cluster.name: as-cluster
node.name: as-node3
node.master: true
node.data: true

network.host: 10.71.X.X

http.port: 9200

discovery.seed_hosts: ["10.71.X.X","10.71.Y.Y","10.71.Z.Z"]

cluster.initial_master_nodes: ["as-node1", "as-node2","as-node3"]

xpack.security.enabled: true
xpack.security.transport.ssl.enabled: true
xpack.security.transport.ssl.verification_mode: full
xpack.security.transport.ssl.keystore.path: config/certs/elastic-certificates.p12
xpack.security.transport.ssl.truststore.path: config/certs/elastic-certificates.p12
xpack.security.http.ssl.enabled: true
xpack.security.http.ssl.keystore.path: config/certs/elastic-certificates.p12
xpack.security.http.ssl.truststore.path: config/certs/elastic-certificates.p12

i am using same CA in all the nodes and providing --ip of respective node while creating certificate. and xpack.security.transport.ssl.verification_mode: full instead of just certificate.

Same configurations for another nodes as well.

5

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.