Unable to add new node in 3 node cluster

Elastic Stack Elasticsearch
1

I have elasticsearch 7.3 installed in 4 nodes, i have successfully configured 3 node cluster but the thing that disturbs me is every time i want to add new node i have to delete old data (i.e /var/lib/elasticsearch/) then after restarting nodes i got the new node in the picture. So this time also i want to add new node but don't want to delete old data, is it because of security, also i am not able to set password for node-4.

es.yml node-1

cluster.name: ElasticsearchStaging
node.name: node-1
path.data: /var/lib/elasticsearch
network.host: 10.128.0.26 
http.port: 9200
discovery.seed_hosts:
   - 10.128.0.26
   - 10.128.0.24
   - 10.128.0.27
   - 10.128.0.30
cluster.initial_master_nodes:
   - node-1
   - node-2
   - node-3
   - node-4
xpack.security.enabled: true
xpack.security.transport.ssl.enabled: true
xpack.security.transport.ssl.verification_mode: certificate
xpack.security.transport.ssl.keystore.path: elastic-certificates.p12
xpack.security.transport.ssl.truststore.path: elastic-certificates.p12

es.yml node-2

cluster.name: ElasticsearchStaging
node.name: node-2
path.data: /var/lib/elasticsearch
network.host: 10.128.0.24 
http.port: 9200
discovery.seed_hosts:
   - 10.128.0.26
   - 10.128.0.24
   - 10.128.0.27
   - 10.128.0.30
cluster.initial_master_nodes:
   - node-1
   - node-2
   - node-3
   - node-4
xpack.security.enabled: true
xpack.security.transport.ssl.enabled: true
xpack.security.transport.ssl.verification_mode: certificate
xpack.security.transport.ssl.keystore.path: elastic-certificates.p12
xpack.security.transport.ssl.truststore.path: elastic-certificates.p12

es.yml node-3

cluster.name: ElasticsearchStaging
node.name: node-3
path.data: /var/lib/elasticsearch
network.host: 10.128.0.27
http.port: 9200
discovery.seed_hosts:
   - 10.128.0.26
   - 10.128.0.24
   - 10.128.0.27
   - 10.128.0.30
cluster.initial_master_nodes:
   - node-1
   - node-2
   - node-3
   - node-4
xpack.security.enabled: true
xpack.security.transport.ssl.enabled: true
xpack.security.transport.ssl.verification_mode: certificate
xpack.security.transport.ssl.keystore.path: elastic-certificates.p12
xpack.security.transport.ssl.truststore.path: elastic-certificates.p12

es.yml node-4

cluster.name: ElasticsearchStaging
node.name: node-4
node.master: true
node.data: false
path.data: /var/lib/elasticsearch
network.host: 10.128.0.30 
http.port: 9200
discovery.seed_hosts:
   - 10.128.0.26
   - 10.128.0.24
   - 10.128.0.27
   - 10.128.0.30
cluster.initial_master_nodes:
   - node-1
   - node-2
   - node-3
   - node-4
xpack.security.enabled: true
xpack.security.transport.ssl.enabled: true
xpack.security.transport.ssl.verification_mode: certificate
xpack.security.transport.ssl.keystore.path: elastic-certificates.p12
xpack.security.transport.ssl.truststore.path: elastic-certificates.p12

update i removed comment all xpack security for now in all yml files

Thanks.

2

node-4 logs

Aug 30 10:31:48 elastic-stage-vyakar-master-1 elasticsearch[8339]: [2019-08-30T10:31:48,095][INFO ][o.e.c.c.JoinHelper       ] [node-4] failed to join {node-2}{Qz0zf9HPTiKZPiX
fBthUdg}{U5TsaIGqQnemP9PQ-48QHg}{10.128.0.24}{10.128.0.24:9300}{dim}{ml.machine_memory=3872587776, ml.max_open_jobs=20, xpack.installed=true} with JoinRequest{sourceNode={node
-4}{VWDFHsUXRdiR380f8TjQEg}{NwL7cQMhTnGnHbNtt_dyBw}{10.128.0.30}{10.128.0.30:9300}{im}{ml.machine_memory=1771679744, xpack.installed=true, ml.max_open_jobs=20}, optionalJoin=O
ptional.empty}
Aug 30 10:31:48 elastic-stage-vyakar-master-1 elasticsearch[8339]: org.elasticsearch.transport.RemoteTransportException: [node-2][10.128.0.24:9300][internal:cluster/coordinati
on/join]
Aug 30 10:31:48 elastic-stage-vyakar-master-1 elasticsearch[8339]: Caused by: java.lang.IllegalStateException: failure when sending a validation request to node
Aug 30 10:31:48 elastic-stage-vyakar-master-1 elasticsearch[8339]:         at org.elasticsearch.cluster.coordination.Coordinator$2.onFailure(Coordinator.java:506) ~[elasticsea
rch-7.3.1.jar:7.3.1]
Aug 30 10:31:48 elastic-stage-vyakar-master-1 elasticsearch[8339]:         at org.elasticsearch.cluster.coordination.JoinHelper$5.handleException(JoinHelper.java:360) ~[elasti
csearch-7.3.1.jar:7.3.1]
Aug 30 10:31:48 elastic-stage-vyakar-master-1 elasticsearch[8339]:         at org.elasticsearch.transport.TransportService$ContextRestoreResponseHandler.handleException(Transp
ortService.java:1111) ~[elasticsearch-7.3.1.jar:7.3.1]
Aug 30 10:31:48 elastic-stage-vyakar-master-1 elasticsearch[8339]:         at org.elasticsearch.transport.InboundHandler.lambda$handleException$2(InboundHandler.java:246) ~[el
asticsearch-7.3.1.jar:7.3.1]
Aug 30 10:31:48 elastic-stage-vyakar-master-1 elasticsearch[8339]:         at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContex
t.java:688) ~[elasticsearch-7.3.1.jar:7.3.1]
Aug 30 10:31:48 elastic-stage-vyakar-master-1 elasticsearch[8339]:         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) ~[?:?]
Aug 30 10:31:48 elastic-stage-vyakar-master-1 elasticsearch[8339]:         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) ~[?:?]
Aug 30 10:31:48 elastic-stage-vyakar-master-1 elasticsearch[8339]:         at java.lang.Thread.run(Thread.java:835) [?:?]
Aug 30 10:31:48 elastic-stage-vyakar-master-1 elasticsearch[8339]: Caused by: org.elasticsearch.transport.RemoteTransportException: [node-4][10.128.0.30:9300][internal:cluster
/coordination/join/validate]
Aug 30 10:31:48 elastic-stage-vyakar-master-1 elasticsearch[8339]: Caused by: org.elasticsearch.cluster.coordination.CoordinationStateRejectedException: join validation on clu
ster state with a different cluster uuid cBaVe_mHT7mmhgB0VBqQGw than local cluster uuid P0xgJFsgQP2NDe1Xgop_tg, rejecting
Aug 30 10:31:48 elastic-stage-vyakar-master-1 elasticsearch[8339]:         at org.elasticsearch.cluster.coordination.JoinHelper.lambda$new$4(JoinHelper.java:148) ~[elasticsear
ch-7.3.1.jar:7.3.1]
Aug 30 10:31:48 elastic-stage-vyakar-master-1 elasticsearch[8339]:         at org.elasticsearch.xpack.security.transport.SecurityServerTransportInterceptor$ProfileSecuredReque
stHandler$1.doRun(SecurityServerTransportInterceptor.java:257) ~[?:?]
Aug 30 10:31:48 elastic-stage-vyakar-master-1 elasticsearch[8339]:         at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elastic
search-7.3.1.jar:7.3.1]
Aug 30 10:31:48 elastic-stage-vyakar-master-1 elasticsearch[8339]:         at org.elasticsearch.xpack.security.transport.SecurityServerTransportInterceptor$ProfileSecuredReque
stHandler.messageReceived(SecurityServerTransportInterceptor.java:315) ~[?:?]
Aug 30 10:31:48 elastic-stage-vyakar-master-1 elasticsearch[8339]:         at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.
java:63) ~[elasticsearch-7.3.1.jar:7.3.1]
Aug 30 10:31:48 elastic-stage-vyakar-master-1 elasticsearch[8339]:         at org.elasticsearch.transport.InboundHandler$RequestHandler.doRun(InboundHandler.java:267) ~[elasti
csearch-7.3.1.jar:7.3.1]
Aug 30 10:31:48 elastic-stage-vyakar-master-1 elasticsearch[8339]:         at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(Th
readContext.java:758) ~[elasticsearch-7.3.1.jar:7.3.1]
Aug 30 10:31:48 elastic-stage-vyakar-master-1 elasticsearch[8339]:         at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elastic
search-7.3.1.jar:7.3.1]
Aug 30 10:31:48 elastic-stage-vyakar-master-1 elasticsearch[8339]:         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) ~[?:?]
Aug 30 10:31:48 elastic-stage-vyakar-master-1 elasticsearch[8339]:         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) ~[?:?]
Aug 30 10:31:48 elastic-stage-vyakar-master-1 elasticsearch[8339]:         at java.lang.Thread.run(Thread.java:835) ~[?:?]
3

This node used to belong to a different cluster, and you cannot move nodes between clusters.

If this happens every time you add a new node then there is something very wrong with your process.

4

Thanks for the reply, But how is it possible i have same cluster.name for all nodes. let me tell you the process i follow.

  1. Create new instance install elasticsearch in it, configure yml file as i already mentioned.
  2. For security i copy the certificate that i created for node-1 paste that certificate in /etc/elasticsearch/ for new node.
  3. add xpack security setting in yml file, and restart all nodes.
  4. Expect new node in the family.
5

Does Elasticsearch run on the new node before you have configured it?

6

yes i created new instance for each node of elasticsearch. Actually i am using google cloud instance. Separate instance or machine for each node i don't know is this is information matters or not but for this node i used system with 1.74GM memory and other 3 nodes are installed in 3.75 GB memory.

node-4 logs

Aug 30 11:38:19 elastic-stage-vyakar-master-1 elasticsearch[9362]: Caused by: java.lang.IllegalStateException: Node is started with node.data=false, but has shard data: [/var/
lib/elasticsearch/nodes/0/indices/1DTqsfUmRPKv2szFCXfc-Q/0]. Use 'elasticsearch-node repurpose' tool to clean up
Aug 30 11:38:19 elastic-stage-vyakar-master-1 elasticsearch[9362]:         at org.elasticsearch.env.NodeEnvironment.ensureNoShardData(NodeEnvironment.java:1081) ~[elasticsearc
h-7.3.1.jar:7.3.1]
Aug 30 11:38:19 elastic-stage-vyakar-master-1 elasticsearch[9362]:         at org.elasticsearch.env.NodeEnvironment.<init>(NodeEnvironment.java:325) ~[elasticsearch-7.3.1.jar:
7.3.1]
Aug 30 11:38:19 elastic-stage-vyakar-master-1 elasticsearch[9362]:         at org.elasticsearch.node.Node.<init>(Node.java:278) ~[elasticsearch-7.3.1.jar:7.3.1]
Aug 30 11:38:19 elastic-stage-vyakar-master-1 elasticsearch[9362]:         at org.elasticsearch.node.Node.<init>(Node.java:258) ~[elasticsearch-7.3.1.jar:7.3.1]
Aug 30 11:38:19 elastic-stage-vyakar-master-1 elasticsearch[9362]:         at org.elasticsearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:221) ~[elasticsearch-7.3.1.jar:7.3.
1]
Aug 30 11:38:19 elastic-stage-vyakar-master-1 elasticsearch[9362]:         at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:221) ~[elasticsearch-7.3.1.jar:7.3.1]
Aug 30 11:38:19 elastic-stage-vyakar-master-1 elasticsearch[9362]:         at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:349) ~[elasticsearch-7.3.1.jar:7.3.1]
Aug 30 11:38:19 elastic-stage-vyakar-master-1 elasticsearch[9362]:         at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:159) ~[elasticsearch-7.3.1.jar:
7.3.1]
Aug 30 11:38:19 elastic-stage-vyakar-master-1 elasticsearch[9362]:         ... 6 more
Aug 30 11:38:19 elastic-stage-vyakar-master-1 systemd[1]: elasticsearch.service: Main process exited, code=exited, status=1/FAILURE
Aug 30 11:38:19 elastic-stage-vyakar-master-1 systemd[1]: elasticsearch.service: Failed with result 'exit-code'.
Aug 30 11:40:35 elastic-stage-vyakar-master-1 systemd[1]: Started Elasticsearch.
Aug 30 11:40:37 elastic-stage-vyakar-master-1 elasticsearch[9506]: OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be
 removed in a future release.
Aug 30 11:40:42 elastic-stage-vyakar-master-1 elasticsearch[9506]: [2019-08-30T11:40:42,230][INFO ][o.e.e.NodeEnvironment    ] [node-4] using [1] data paths, mounts [[/ (/dev/
sda1)]], net usable_space [7.3gb], net total_space [9.5gb], types [ext4]
Aug 30 11:40:42 elastic-stage-vyakar-master-1 elasticsearch[9506]: [2019-08-30T11:40:42,243][INFO ][o.e.e.NodeEnvironment    ] [node-4] heap size [1015.6mb], compressed ordina
ry object pointers [true]
Aug 30 11:40:42 elastic-stage-vyakar-master-1 elasticsearch[9506]: [2019-08-30T11:40:42,325][WARN ][o.e.b.ElasticsearchUncaughtExceptionHandler] [node-4] uncaught exception in
 thread [main]
Aug 30 11:40:42 elastic-stage-vyakar-master-1 elasticsearch[9506]: org.elasticsearch.bootstrap.StartupException: java.lang.IllegalStateException: Node is started with node.dat
a=false, but has shard data: [/var/lib/elasticsearch/nodes/0/indices/1DTqsfUmRPKv2szFCXfc-Q/0]. Use 'elasticsearch-node repurpose' tool to clean up
Aug 30 11:40:42 elastic-stage-vyakar-master-1 elasticsearch[9506]:         at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:163) ~[elasticsearch-7.3.1.jar:
7.3.1]
Aug 30 11:40:42 elastic-stage-vyakar-master-1 elasticsearch[9506]:         at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:150) ~[elasticsearch-7.3.1.j
ar:7.3.1]
Aug 30 11:40:42 elastic-stage-vyakar-master-1 elasticsearch[9506]:         at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:86) ~[elastics
earch-7.3.1.jar:7.3.1]
Aug 30 11:40:42 elastic-stage-vyakar-master-1 elasticsearch[9506]:         at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:124) ~[elasticsearch-cli-7.3.
1.jar:7.3.1]
Aug 30 11:40:42 elastic-stage-vyakar-master-1 elasticsearch[9506]:         at org.elasticsearch.cli.Command.main(Command.java:90) ~[elasticsearch-cli-7.3.1.jar:7.3.1]
Aug 30 11:40:42 elastic-stage-vyakar-master-1 elasticsearch[9506]:         at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:115) ~[elasticsearch-7.3.1.jar:
7.3.1]
7

It looks like you are trying to configure the node after it's already started up once. It's important to configure it correctly before running it for the first time.