I have elasticsearch 7.3 installed in 4 nodes, i have successfully configured 3 node cluster but the thing that disturbs me is every time i want to add new node i have to delete old data (i.e /var/lib/elasticsearch/) then after restarting nodes i got the new node in the picture. So this time also i want to add new node but don't want to delete old data, is it because of security, also i am not able to set password for node-4.
Aug 30 10:31:48 elastic-stage-vyakar-master-1 elasticsearch[8339]: [2019-08-30T10:31:48,095][INFO ][o.e.c.c.JoinHelper ] [node-4] failed to join {node-2}{Qz0zf9HPTiKZPiX
fBthUdg}{U5TsaIGqQnemP9PQ-48QHg}{10.128.0.24}{10.128.0.24:9300}{dim}{ml.machine_memory=3872587776, ml.max_open_jobs=20, xpack.installed=true} with JoinRequest{sourceNode={node
-4}{VWDFHsUXRdiR380f8TjQEg}{NwL7cQMhTnGnHbNtt_dyBw}{10.128.0.30}{10.128.0.30:9300}{im}{ml.machine_memory=1771679744, xpack.installed=true, ml.max_open_jobs=20}, optionalJoin=O
ptional.empty}
Aug 30 10:31:48 elastic-stage-vyakar-master-1 elasticsearch[8339]: org.elasticsearch.transport.RemoteTransportException: [node-2][10.128.0.24:9300][internal:cluster/coordinati
on/join]
Aug 30 10:31:48 elastic-stage-vyakar-master-1 elasticsearch[8339]: Caused by: java.lang.IllegalStateException: failure when sending a validation request to node
Aug 30 10:31:48 elastic-stage-vyakar-master-1 elasticsearch[8339]: at org.elasticsearch.cluster.coordination.Coordinator$2.onFailure(Coordinator.java:506) ~[elasticsea
rch-7.3.1.jar:7.3.1]
Aug 30 10:31:48 elastic-stage-vyakar-master-1 elasticsearch[8339]: at org.elasticsearch.cluster.coordination.JoinHelper$5.handleException(JoinHelper.java:360) ~[elasti
csearch-7.3.1.jar:7.3.1]
Aug 30 10:31:48 elastic-stage-vyakar-master-1 elasticsearch[8339]: at org.elasticsearch.transport.TransportService$ContextRestoreResponseHandler.handleException(Transp
ortService.java:1111) ~[elasticsearch-7.3.1.jar:7.3.1]
Aug 30 10:31:48 elastic-stage-vyakar-master-1 elasticsearch[8339]: at org.elasticsearch.transport.InboundHandler.lambda$handleException$2(InboundHandler.java:246) ~[el
asticsearch-7.3.1.jar:7.3.1]
Aug 30 10:31:48 elastic-stage-vyakar-master-1 elasticsearch[8339]: at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContex
t.java:688) ~[elasticsearch-7.3.1.jar:7.3.1]
Aug 30 10:31:48 elastic-stage-vyakar-master-1 elasticsearch[8339]: at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) ~[?:?]
Aug 30 10:31:48 elastic-stage-vyakar-master-1 elasticsearch[8339]: at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) ~[?:?]
Aug 30 10:31:48 elastic-stage-vyakar-master-1 elasticsearch[8339]: at java.lang.Thread.run(Thread.java:835) [?:?]
Aug 30 10:31:48 elastic-stage-vyakar-master-1 elasticsearch[8339]: Caused by: org.elasticsearch.transport.RemoteTransportException: [node-4][10.128.0.30:9300][internal:cluster
/coordination/join/validate]
Aug 30 10:31:48 elastic-stage-vyakar-master-1 elasticsearch[8339]: Caused by: org.elasticsearch.cluster.coordination.CoordinationStateRejectedException: join validation on clu
ster state with a different cluster uuid cBaVe_mHT7mmhgB0VBqQGw than local cluster uuid P0xgJFsgQP2NDe1Xgop_tg, rejecting
Aug 30 10:31:48 elastic-stage-vyakar-master-1 elasticsearch[8339]: at org.elasticsearch.cluster.coordination.JoinHelper.lambda$new$4(JoinHelper.java:148) ~[elasticsear
ch-7.3.1.jar:7.3.1]
Aug 30 10:31:48 elastic-stage-vyakar-master-1 elasticsearch[8339]: at org.elasticsearch.xpack.security.transport.SecurityServerTransportInterceptor$ProfileSecuredReque
stHandler$1.doRun(SecurityServerTransportInterceptor.java:257) ~[?:?]
Aug 30 10:31:48 elastic-stage-vyakar-master-1 elasticsearch[8339]: at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elastic
search-7.3.1.jar:7.3.1]
Aug 30 10:31:48 elastic-stage-vyakar-master-1 elasticsearch[8339]: at org.elasticsearch.xpack.security.transport.SecurityServerTransportInterceptor$ProfileSecuredReque
stHandler.messageReceived(SecurityServerTransportInterceptor.java:315) ~[?:?]
Aug 30 10:31:48 elastic-stage-vyakar-master-1 elasticsearch[8339]: at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.
java:63) ~[elasticsearch-7.3.1.jar:7.3.1]
Aug 30 10:31:48 elastic-stage-vyakar-master-1 elasticsearch[8339]: at org.elasticsearch.transport.InboundHandler$RequestHandler.doRun(InboundHandler.java:267) ~[elasti
csearch-7.3.1.jar:7.3.1]
Aug 30 10:31:48 elastic-stage-vyakar-master-1 elasticsearch[8339]: at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(Th
readContext.java:758) ~[elasticsearch-7.3.1.jar:7.3.1]
Aug 30 10:31:48 elastic-stage-vyakar-master-1 elasticsearch[8339]: at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elastic
search-7.3.1.jar:7.3.1]
Aug 30 10:31:48 elastic-stage-vyakar-master-1 elasticsearch[8339]: at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) ~[?:?]
Aug 30 10:31:48 elastic-stage-vyakar-master-1 elasticsearch[8339]: at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) ~[?:?]
Aug 30 10:31:48 elastic-stage-vyakar-master-1 elasticsearch[8339]: at java.lang.Thread.run(Thread.java:835) ~[?:?]
yes i created new instance for each node of elasticsearch. Actually i am using google cloud instance. Separate instance or machine for each node i don't know is this is information matters or not but for this node i used system with 1.74GM memory and other 3 nodes are installed in 3.75 GB memory.
node-4 logs
Aug 30 11:38:19 elastic-stage-vyakar-master-1 elasticsearch[9362]: Caused by: java.lang.IllegalStateException: Node is started with node.data=false, but has shard data: [/var/
lib/elasticsearch/nodes/0/indices/1DTqsfUmRPKv2szFCXfc-Q/0]. Use 'elasticsearch-node repurpose' tool to clean up
Aug 30 11:38:19 elastic-stage-vyakar-master-1 elasticsearch[9362]: at org.elasticsearch.env.NodeEnvironment.ensureNoShardData(NodeEnvironment.java:1081) ~[elasticsearc
h-7.3.1.jar:7.3.1]
Aug 30 11:38:19 elastic-stage-vyakar-master-1 elasticsearch[9362]: at org.elasticsearch.env.NodeEnvironment.<init>(NodeEnvironment.java:325) ~[elasticsearch-7.3.1.jar:
7.3.1]
Aug 30 11:38:19 elastic-stage-vyakar-master-1 elasticsearch[9362]: at org.elasticsearch.node.Node.<init>(Node.java:278) ~[elasticsearch-7.3.1.jar:7.3.1]
Aug 30 11:38:19 elastic-stage-vyakar-master-1 elasticsearch[9362]: at org.elasticsearch.node.Node.<init>(Node.java:258) ~[elasticsearch-7.3.1.jar:7.3.1]
Aug 30 11:38:19 elastic-stage-vyakar-master-1 elasticsearch[9362]: at org.elasticsearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:221) ~[elasticsearch-7.3.1.jar:7.3.
1]
Aug 30 11:38:19 elastic-stage-vyakar-master-1 elasticsearch[9362]: at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:221) ~[elasticsearch-7.3.1.jar:7.3.1]
Aug 30 11:38:19 elastic-stage-vyakar-master-1 elasticsearch[9362]: at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:349) ~[elasticsearch-7.3.1.jar:7.3.1]
Aug 30 11:38:19 elastic-stage-vyakar-master-1 elasticsearch[9362]: at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:159) ~[elasticsearch-7.3.1.jar:
7.3.1]
Aug 30 11:38:19 elastic-stage-vyakar-master-1 elasticsearch[9362]: ... 6 more
Aug 30 11:38:19 elastic-stage-vyakar-master-1 systemd[1]: elasticsearch.service: Main process exited, code=exited, status=1/FAILURE
Aug 30 11:38:19 elastic-stage-vyakar-master-1 systemd[1]: elasticsearch.service: Failed with result 'exit-code'.
Aug 30 11:40:35 elastic-stage-vyakar-master-1 systemd[1]: Started Elasticsearch.
Aug 30 11:40:37 elastic-stage-vyakar-master-1 elasticsearch[9506]: OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be
removed in a future release.
Aug 30 11:40:42 elastic-stage-vyakar-master-1 elasticsearch[9506]: [2019-08-30T11:40:42,230][INFO ][o.e.e.NodeEnvironment ] [node-4] using [1] data paths, mounts [[/ (/dev/
sda1)]], net usable_space [7.3gb], net total_space [9.5gb], types [ext4]
Aug 30 11:40:42 elastic-stage-vyakar-master-1 elasticsearch[9506]: [2019-08-30T11:40:42,243][INFO ][o.e.e.NodeEnvironment ] [node-4] heap size [1015.6mb], compressed ordina
ry object pointers [true]
Aug 30 11:40:42 elastic-stage-vyakar-master-1 elasticsearch[9506]: [2019-08-30T11:40:42,325][WARN ][o.e.b.ElasticsearchUncaughtExceptionHandler] [node-4] uncaught exception in
thread [main]
Aug 30 11:40:42 elastic-stage-vyakar-master-1 elasticsearch[9506]: org.elasticsearch.bootstrap.StartupException: java.lang.IllegalStateException: Node is started with node.dat
a=false, but has shard data: [/var/lib/elasticsearch/nodes/0/indices/1DTqsfUmRPKv2szFCXfc-Q/0]. Use 'elasticsearch-node repurpose' tool to clean up
Aug 30 11:40:42 elastic-stage-vyakar-master-1 elasticsearch[9506]: at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:163) ~[elasticsearch-7.3.1.jar:
7.3.1]
Aug 30 11:40:42 elastic-stage-vyakar-master-1 elasticsearch[9506]: at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:150) ~[elasticsearch-7.3.1.j
ar:7.3.1]
Aug 30 11:40:42 elastic-stage-vyakar-master-1 elasticsearch[9506]: at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:86) ~[elastics
earch-7.3.1.jar:7.3.1]
Aug 30 11:40:42 elastic-stage-vyakar-master-1 elasticsearch[9506]: at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:124) ~[elasticsearch-cli-7.3.
1.jar:7.3.1]
Aug 30 11:40:42 elastic-stage-vyakar-master-1 elasticsearch[9506]: at org.elasticsearch.cli.Command.main(Command.java:90) ~[elasticsearch-cli-7.3.1.jar:7.3.1]
Aug 30 11:40:42 elastic-stage-vyakar-master-1 elasticsearch[9506]: at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:115) ~[elasticsearch-7.3.1.jar:
7.3.1]
It looks like you are trying to configure the node after it's already started up once. It's important to configure it correctly before running it for the first time.
Apache, Apache Lucene, Apache Hadoop, Hadoop, HDFS and the yellow elephant
logo are trademarks of the
Apache Software Foundation
in the United States and/or other countries.