The third node cannot join the cluster (ES-7.1.0)

Elastic Stack Elasticsearch
1

Hi,
I am using the ES 7.1.0 with below config. The second node can join the cluster successfully, but when I start to the third node, it failed to join the cluster due to below error.

`---

cluster.name: dev-elasticsearch
node.name: sgt-001
path.data: "/var/lib/elasticsearch"
path.logs: "/var/log/elasticsearch"
node.master: true
node.data: true
network.host: 0.0.0.0
discovery.zen.ping.unicast.hosts:
- 192.168.0.81
- 192.168.0.82
- 192.168.0.83
discovery.zen.minimum_master_nodes: 2
cluster.initial_master_nodes:
- 192.168.0.81
- 192.168.0.82
- 192.168.0.83
xpack.security.enabled: true
xpack.security.transport.ssl.enabled: false

`

error log.
[2019-05-30T22:54:09,216][WARN ][o.e.c.l.LogConfigurator ] [sgt-002] Some logging configurations have %marker but don't have %node_name. We will automatically add %node_name to the pattern to ease the migration for users who customize log4j2.properties but will stop this behavior in 7.0. You should manually replace%node_namewith[%node_name]%marker in these locations: /etc/elasticsearch/log4j2.properties [2019-05-30T22:54:09,486][INFO ][o.e.e.NodeEnvironment ] [sgt-002] using [1] data paths, mounts [[/ (rootfs)]], net usable_space [42.7gb], net total_space [49.9gb], types [rootfs] [2019-05-30T22:54:09,487][INFO ][o.e.e.NodeEnvironment ] [sgt-002] heap size [3.9gb], compressed ordinary object pointers [true] [2019-05-30T22:54:09,489][INFO ][o.e.n.Node ] [sgt-002] node name [sgt-002], node ID [jkNZho44TyKzDgV36YKz1A], cluster name [dev-elasticsearch] [2019-05-30T22:54:09,489][INFO ][o.e.n.Node ] [sgt-002] version[7.1.0], pid[27954], build[default/tar/606a173/2019-05-16T00:43:15.323135Z], OS[Linux/3.10.0-957.el7.x86_64/amd64], JVM[Oracle Corporation/OpenJDK 64-Bit Server VM/12.0.1/12.0.1+12] [2019-05-30T22:54:09,489][INFO ][o.e.n.Node ] [sgt-002] JVM home [/usr/share/elasticsearch-7.1.0/jdk] [2019-05-30T22:54:09,489][INFO ][o.e.n.Node ] [sgt-002] JVM arguments [-Xms4096m, -Xmx4096m, -XX:+UseConcMarkSweepGC, -XX:CMSInitiatingOccupancyFraction=75, -XX:+UseCMSInitiatingOccupancyOnly, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -XX:-OmitStackTraceInFastThrow, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -XX:+HeapDumpOnOutOfMemoryError, -Dio.netty.allocator.type=pooled, -Des.path.home=/usr/share/elasticsearch, -Des.path.conf=/etc/elasticsearch, -Des.distribution.flavor=default, -Des.distribution.type=tar, -Des.bundled_jdk=true] ...(delete some lines) [2019-05-30T22:54:10,952][INFO ][o.e.p.PluginsService ] [sgt-002] no plugins loaded [2019-05-30T22:54:14,859][INFO ][o.e.x.m.p.l.CppLogMessageHandler] [sgt-002] [controller/28041] [Main.cc@109] controller (64 bit): Version 7.1.0 (Build a8ee6de8087169) Copyright (c) 2019 Elasticsearch BV [2019-05-30T22:54:15,190][DEBUG][o.e.a.ActionModule ] [sgt-002] Using REST wrapper from plugin org.elasticsearch.xpack.security.Security [2019-05-30T22:54:15,399][INFO ][o.e.d.DiscoveryModule ] [sgt-002] using discovery type [zen] and seed hosts providers [settings] [2019-05-30T22:54:16,110][INFO ][o.e.n.Node ] [sgt-002] initialized [2019-05-30T22:54:16,110][INFO ][o.e.n.Node ] [sgt-002] starting ... [2019-05-30T22:54:16,229][INFO ][o.e.t.TransportService ] [sgt-002] publish_address {192.168.0.82:9300}, bound_addresses {0.0.0.0:9300} [2019-05-30T22:54:16,236][INFO ][o.e.b.BootstrapChecks ] [sgt-002] bound or publishing to a non-loopback address, enforcing bootstrap checks [2019-05-30T22:54:16,496][INFO ][o.e.c.c.Coordinator ] [sgt-002] setting initial configuration to VotingConfiguration{jkNZho44TyKzDgV36YKz1A,{bootstrap-placeholder}-192.168.0.83,BFXYE9tzRx-2J6LJX8O6Hw} [2019-05-30T22:54:16,637][INFO ][o.e.c.c.JoinHelper ] [sgt-002] failed to join {sgt-003}{M5EJGeXESjWyjyAvSEfqbg}{zRFod2KFSS-Ot6I28S58eg}{192.168.0.83}{192.168.0.83:9300}{ml.machine_memory=8201474048, ml.max_open_jobs=20, xpack.installed=true} with JoinRequest{sourceNode={sgt-002}{jkNZho44TyKzDgV36YKz1A}{bTckyMyrQeqo5trP4SRy7g}{192.168.0.82}{192.168.0.82:9300}{ml.machine_memory=8201474048, xpack.installed=true, ml.max_open_jobs=20}, optionalJoin=Optional[Join{term=1, lastAcceptedTerm=0, lastAcceptedVersion=0, sourceNode={sgt-002}{jkNZho44TyKzDgV36YKz1A}{bTckyMyrQeqo5trP4SRy7g}{192.168.0.82}{192.168.0.82:9300}{ml.machine_memory=8201474048, xpack.installed=true, ml.max_open_jobs=20}, targetNode={sgt-003}{M5EJGeXESjWyjyAvSEfqbg}{zRFod2KFSS-Ot6I28S58eg}{192.168.0.83}{192.168.0.83:9300}{ml.machine_memory=8201474048, ml.max_open_jobs=20, xpack.installed=true}}]} org.elasticsearch.transport.RemoteTransportException: [sgt-003][192.168.0.83:9300][internal:cluster/coordination/join] Caused by: java.lang.IllegalStateException: Transport TLS ([xpack.security.transport.ssl.enabled]) is required for license type [basic] when security is enabled at org.elasticsearch.xpack.security.Security$ValidateTLSOnJoin.accept(Security.java:993) ~[?:?] at org.elasticsearch.xpack.security.Security$ValidateTLSOnJoin.accept(Security.java:976) ~[?:?] at java.util.function.BiConsumer.lambda$andThen$0(BiConsumer.java:71) ~[?:?] at java.util.function.BiConsumer.lambda$andThen$0(BiConsumer.java:71) ~[?:?] at java.util.function.BiConsumer.lambda$andThen$0(BiConsumer.java:71) ~[?:?] ....

2

I think it is because you enabled xpack which requires license.
You should remove these 2 lines for all config

xpack.security.enabled: true
xpack.security.transport.ssl.enabled: false

Also discovery.zen.ping.unicast.hosts have changed to discovery.seed_hosts in version 7

1 Like
3

I need to enable xpack.security.enabled and disable xpack.security.transport.ssl.enabled. So I modify the configuration as below and tried again. still get the same error

---
cluster.name: dev-elasticsearch
node.name: sgt-001
path.data: "/var/lib/elasticsearch"
path.logs: "/var/log/elasticsearch"
node.master: true
node.data: true
network.host: 0.0.0.0
discovery.seed_hosts:
- 192.168.0.81
- 192.168.0.82
- 192.168.0.83
cluster.initial_master_nodes:
- 192.168.0.81
- 192.168.0.82
- 192.168.0.83
xpack.security.enabled: true
xpack.security.transport.ssl.enabled: false
4

Hi ellison001
Do you mean you have license? Thats why you have to enable xpack?

Have you tried this configuration without xpack and does it work?

cluster.name: dev-elasticsearch
node.name: sgt-001
path.data: "/var/lib/elasticsearch"
path.logs: "/var/log/elasticsearch"
node.master: true
node.data: true
network.host: 0.0.0.0
discovery.seed_hosts:
- 192.168.0.81
- 192.168.0.82
- 192.168.0.83
cluster.initial_master_nodes:
- 192.168.0.81
- 192.168.0.82
- 192.168.0.83
5

This configuration doesn't make sense. Without TLS everything is in plain text, so there is no point in using security. This is what the exception message says:

Can you explain what you're trying to achieve here? Why do you want to avoid setting up TLS?

6

Hi wk3000sg, I don't purchase the license. I use the basic license. since ES announce that the limited security feature for x-pack is free with ES6.8.0 and ES7.1.0.

I just tried to remove the x-pack security settings. all the three node can join the cluster without any problem.

7

Hi David,
The ES I set up is only for internal use. I put the ES node behind a load balancer, and do the SSL offline via load balancer. That's why I just want to enable the basic authentication via x-pack without transport SSL enabled.

From the ES7.1 document, it doesn't mention that xpack.security.enabled and xpack.security.transport.ssl.enabled have some restrictions. like if xpack.security.enabled set to true, xpack.security.transport.ssl.enabled must set to true.

8

Hi David,

I too have a similar scenario, I have my Elastic Stack behind a reverse proxy and when I enable xpack it asks to enable TLS/SSL. But in the official documentation its not mentioned. Also, why should I enable SSL when my ES cluster is running behind a reverse proxy.

9

It worked for me when I made the below entry in elasticsearch.yml file and I have not configured any certificate and it worked.

xpack.security.enabled: true
xpack.security.transport.ssl.enabled: true

10

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.