The third node cannot join the cluster (ES-7.1.0)

Hi,
I am using the ES 7.1.0 with below config. The second node can join the cluster successfully, but when I start to the third node, it failed to join the cluster due to below error.

`---

cluster.name: dev-elasticsearch
node.name: sgt-001
path.data: "/var/lib/elasticsearch"
path.logs: "/var/log/elasticsearch"
node.master: true
node.data: true
network.host: 0.0.0.0
discovery.zen.ping.unicast.hosts:
- 192.168.0.81
- 192.168.0.82
- 192.168.0.83
discovery.zen.minimum_master_nodes: 2
cluster.initial_master_nodes:
- 192.168.0.81
- 192.168.0.82
- 192.168.0.83
xpack.security.enabled: true
xpack.security.transport.ssl.enabled: false

`

error log.
[2019-05-30T22:54:09,216][WARN ][o.e.c.l.LogConfigurator ] [sgt-002] Some logging configurations have %marker but don't have %node_name. We will automatically add %node_name to the pattern to ease the migration for users who customize log4j2.properties but will stop this behavior in 7.0. You should manually replace%node_namewith[%node_name]%marker in these locations: /etc/elasticsearch/log4j2.properties [2019-05-30T22:54:09,486][INFO ][o.e.e.NodeEnvironment ] [sgt-002] using [1] data paths, mounts [[/ (rootfs)]], net usable_space [42.7gb], net total_space [49.9gb], types [rootfs] [2019-05-30T22:54:09,487][INFO ][o.e.e.NodeEnvironment ] [sgt-002] heap size [3.9gb], compressed ordinary object pointers [true] [2019-05-30T22:54:09,489][INFO ][o.e.n.Node ] [sgt-002] node name [sgt-002], node ID [jkNZho44TyKzDgV36YKz1A], cluster name [dev-elasticsearch] [2019-05-30T22:54:09,489][INFO ][o.e.n.Node ] [sgt-002] version[7.1.0], pid[27954], build[default/tar/606a173/2019-05-16T00:43:15.323135Z], OS[Linux/3.10.0-957.el7.x86_64/amd64], JVM[Oracle Corporation/OpenJDK 64-Bit Server VM/12.0.1/12.0.1+12] [2019-05-30T22:54:09,489][INFO ][o.e.n.Node ] [sgt-002] JVM home [/usr/share/elasticsearch-7.1.0/jdk] [2019-05-30T22:54:09,489][INFO ][o.e.n.Node ] [sgt-002] JVM arguments [-Xms4096m, -Xmx4096m, -XX:+UseConcMarkSweepGC, -XX:CMSInitiatingOccupancyFraction=75, -XX:+UseCMSInitiatingOccupancyOnly, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -XX:-OmitStackTraceInFastThrow, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -XX:+HeapDumpOnOutOfMemoryError, -Dio.netty.allocator.type=pooled, -Des.path.home=/usr/share/elasticsearch, -Des.path.conf=/etc/elasticsearch, -Des.distribution.flavor=default, -Des.distribution.type=tar, -Des.bundled_jdk=true] ...(delete some lines) [2019-05-30T22:54:10,952][INFO ][o.e.p.PluginsService ] [sgt-002] no plugins loaded [2019-05-30T22:54:14,859][INFO ][o.e.x.m.p.l.CppLogMessageHandler] [sgt-002] [controller/28041] [Main.cc@109] controller (64 bit): Version 7.1.0 (Build a8ee6de8087169) Copyright (c) 2019 Elasticsearch BV [2019-05-30T22:54:15,190][DEBUG][o.e.a.ActionModule ] [sgt-002] Using REST wrapper from plugin org.elasticsearch.xpack.security.Security [2019-05-30T22:54:15,399][INFO ][o.e.d.DiscoveryModule ] [sgt-002] using discovery type [zen] and seed hosts providers [settings] [2019-05-30T22:54:16,110][INFO ][o.e.n.Node ] [sgt-002] initialized [2019-05-30T22:54:16,110][INFO ][o.e.n.Node ] [sgt-002] starting ... [2019-05-30T22:54:16,229][INFO ][o.e.t.TransportService ] [sgt-002] publish_address {192.168.0.82:9300}, bound_addresses {0.0.0.0:9300} [2019-05-30T22:54:16,236][INFO ][o.e.b.BootstrapChecks ] [sgt-002] bound or publishing to a non-loopback address, enforcing bootstrap checks [2019-05-30T22:54:16,496][INFO ][o.e.c.c.Coordinator ] [sgt-002] setting initial configuration to VotingConfiguration{jkNZho44TyKzDgV36YKz1A,{bootstrap-placeholder}-192.168.0.83,BFXYE9tzRx-2J6LJX8O6Hw} [2019-05-30T22:54:16,637][INFO ][o.e.c.c.JoinHelper ] [sgt-002] failed to join {sgt-003}{M5EJGeXESjWyjyAvSEfqbg}{zRFod2KFSS-Ot6I28S58eg}{192.168.0.83}{192.168.0.83:9300}{ml.machine_memory=8201474048, ml.max_open_jobs=20, xpack.installed=true} with JoinRequest{sourceNode={sgt-002}{jkNZho44TyKzDgV36YKz1A}{bTckyMyrQeqo5trP4SRy7g}{192.168.0.82}{192.168.0.82:9300}{ml.machine_memory=8201474048, xpack.installed=true, ml.max_open_jobs=20}, optionalJoin=Optional[Join{term=1, lastAcceptedTerm=0, lastAcceptedVersion=0, sourceNode={sgt-002}{jkNZho44TyKzDgV36YKz1A}{bTckyMyrQeqo5trP4SRy7g}{192.168.0.82}{192.168.0.82:9300}{ml.machine_memory=8201474048, xpack.installed=true, ml.max_open_jobs=20}, targetNode={sgt-003}{M5EJGeXESjWyjyAvSEfqbg}{zRFod2KFSS-Ot6I28S58eg}{192.168.0.83}{192.168.0.83:9300}{ml.machine_memory=8201474048, ml.max_open_jobs=20, xpack.installed=true}}]} org.elasticsearch.transport.RemoteTransportException: [sgt-003][192.168.0.83:9300][internal:cluster/coordination/join] Caused by: java.lang.IllegalStateException: Transport TLS ([xpack.security.transport.ssl.enabled]) is required for license type [basic] when security is enabled at org.elasticsearch.xpack.security.Security$ValidateTLSOnJoin.accept(Security.java:993) ~[?:?] at org.elasticsearch.xpack.security.Security$ValidateTLSOnJoin.accept(Security.java:976) ~[?:?] at java.util.function.BiConsumer.lambda$andThen$0(BiConsumer.java:71) ~[?:?] at java.util.function.BiConsumer.lambda$andThen$0(BiConsumer.java:71) ~[?:?] at java.util.function.BiConsumer.lambda$andThen$0(BiConsumer.java:71) ~[?:?] ....

I think it is because you enabled xpack which requires license.
You should remove these 2 lines for all config

xpack.security.enabled: true
xpack.security.transport.ssl.enabled: false

Also discovery.zen.ping.unicast.hosts have changed to discovery.seed_hosts in version 7

1 Like

I need to enable xpack.security.enabled and disable xpack.security.transport.ssl.enabled. So I modify the configuration as below and tried again. still get the same error

---
cluster.name: dev-elasticsearch
node.name: sgt-001
path.data: "/var/lib/elasticsearch"
path.logs: "/var/log/elasticsearch"
node.master: true
node.data: true
network.host: 0.0.0.0
discovery.seed_hosts:
- 192.168.0.81
- 192.168.0.82
- 192.168.0.83
cluster.initial_master_nodes:
- 192.168.0.81
- 192.168.0.82
- 192.168.0.83
xpack.security.enabled: true
xpack.security.transport.ssl.enabled: false

Hi ellison001
Do you mean you have license? Thats why you have to enable xpack?

Have you tried this configuration without xpack and does it work?

cluster.name: dev-elasticsearch
node.name: sgt-001
path.data: "/var/lib/elasticsearch"
path.logs: "/var/log/elasticsearch"
node.master: true
node.data: true
network.host: 0.0.0.0
discovery.seed_hosts:
- 192.168.0.81
- 192.168.0.82
- 192.168.0.83
cluster.initial_master_nodes:
- 192.168.0.81
- 192.168.0.82
- 192.168.0.83

This configuration doesn't make sense. Without TLS everything is in plain text, so there is no point in using security. This is what the exception message says:

Can you explain what you're trying to achieve here? Why do you want to avoid setting up TLS?

Hi wk3000sg, I don't purchase the license. I use the basic license. since ES announce that the limited security feature for x-pack is free with ES6.8.0 and ES7.1.0.

I just tried to remove the x-pack security settings. all the three node can join the cluster without any problem.

Hi David,
The ES I set up is only for internal use. I put the ES node behind a load balancer, and do the SSL offline via load balancer. That's why I just want to enable the basic authentication via x-pack without transport SSL enabled.

From the ES7.1 document, it doesn't mention that xpack.security.enabled and xpack.security.transport.ssl.enabled have some restrictions. like if xpack.security.enabled set to true, xpack.security.transport.ssl.enabled must set to true.

Hi David,

I too have a similar scenario, I have my Elastic Stack behind a reverse proxy and when I enable xpack it asks to enable TLS/SSL. But in the official documentation its not mentioned. Also, why should I enable SSL when my ES cluster is running behind a reverse proxy.

It worked for me when I made the below entry in elasticsearch.yml file and I have not configured any certificate and it worked.

xpack.security.enabled: true
xpack.security.transport.ssl.enabled: true

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.