Unable to add new node in 3 node cluster

Ayush_Verma · August 30, 2019, 10:22am

I have elasticsearch 7.3 installed in 4 nodes, i have successfully configured 3 node cluster but the thing that disturbs me is every time i want to add new node i have to delete old data (i.e /var/lib/elasticsearch/) then after restarting nodes i got the new node in the picture. So this time also i want to add new node but don't want to delete old data, is it because of security, also i am not able to set password for node-4.

es.yml node-1

cluster.name: ElasticsearchStaging
node.name: node-1
path.data: /var/lib/elasticsearch
network.host: 10.128.0.26 
http.port: 9200
discovery.seed_hosts:
   - 10.128.0.26
   - 10.128.0.24
   - 10.128.0.27
   - 10.128.0.30
cluster.initial_master_nodes:
   - node-1
   - node-2
   - node-3
   - node-4
xpack.security.enabled: true
xpack.security.transport.ssl.enabled: true
xpack.security.transport.ssl.verification_mode: certificate
xpack.security.transport.ssl.keystore.path: elastic-certificates.p12
xpack.security.transport.ssl.truststore.path: elastic-certificates.p12

es.yml node-2

cluster.name: ElasticsearchStaging
node.name: node-2
path.data: /var/lib/elasticsearch
network.host: 10.128.0.24 
http.port: 9200
discovery.seed_hosts:
   - 10.128.0.26
   - 10.128.0.24
   - 10.128.0.27
   - 10.128.0.30
cluster.initial_master_nodes:
   - node-1
   - node-2
   - node-3
   - node-4
xpack.security.enabled: true
xpack.security.transport.ssl.enabled: true
xpack.security.transport.ssl.verification_mode: certificate
xpack.security.transport.ssl.keystore.path: elastic-certificates.p12
xpack.security.transport.ssl.truststore.path: elastic-certificates.p12

es.yml node-3

cluster.name: ElasticsearchStaging
node.name: node-3
path.data: /var/lib/elasticsearch
network.host: 10.128.0.27
http.port: 9200
discovery.seed_hosts:
   - 10.128.0.26
   - 10.128.0.24
   - 10.128.0.27
   - 10.128.0.30
cluster.initial_master_nodes:
   - node-1
   - node-2
   - node-3
   - node-4
xpack.security.enabled: true
xpack.security.transport.ssl.enabled: true
xpack.security.transport.ssl.verification_mode: certificate
xpack.security.transport.ssl.keystore.path: elastic-certificates.p12
xpack.security.transport.ssl.truststore.path: elastic-certificates.p12

es.yml node-4

cluster.name: ElasticsearchStaging
node.name: node-4
node.master: true
node.data: false
path.data: /var/lib/elasticsearch
network.host: 10.128.0.30 
http.port: 9200
discovery.seed_hosts:
   - 10.128.0.26
   - 10.128.0.24
   - 10.128.0.27
   - 10.128.0.30
cluster.initial_master_nodes:
   - node-1
   - node-2
   - node-3
   - node-4
xpack.security.enabled: true
xpack.security.transport.ssl.enabled: true
xpack.security.transport.ssl.verification_mode: certificate
xpack.security.transport.ssl.keystore.path: elastic-certificates.p12
xpack.security.transport.ssl.truststore.path: elastic-certificates.p12

update i removed comment all xpack security for now in all yml files

Thanks.

Ayush_Verma · August 30, 2019, 10:38am

node-4 logs

Aug 30 10:31:48 elastic-stage-vyakar-master-1 elasticsearch[8339]: [2019-08-30T10:31:48,095][INFO ][o.e.c.c.JoinHelper       ] [node-4] failed to join {node-2}{Qz0zf9HPTiKZPiX
fBthUdg}{U5TsaIGqQnemP9PQ-48QHg}{10.128.0.24}{10.128.0.24:9300}{dim}{ml.machine_memory=3872587776, ml.max_open_jobs=20, xpack.installed=true} with JoinRequest{sourceNode={node
-4}{VWDFHsUXRdiR380f8TjQEg}{NwL7cQMhTnGnHbNtt_dyBw}{10.128.0.30}{10.128.0.30:9300}{im}{ml.machine_memory=1771679744, xpack.installed=true, ml.max_open_jobs=20}, optionalJoin=O
ptional.empty}
Aug 30 10:31:48 elastic-stage-vyakar-master-1 elasticsearch[8339]: org.elasticsearch.transport.RemoteTransportException: [node-2][10.128.0.24:9300][internal:cluster/coordinati
on/join]
Aug 30 10:31:48 elastic-stage-vyakar-master-1 elasticsearch[8339]: Caused by: java.lang.IllegalStateException: failure when sending a validation request to node
Aug 30 10:31:48 elastic-stage-vyakar-master-1 elasticsearch[8339]:         at org.elasticsearch.cluster.coordination.Coordinator$2.onFailure(Coordinator.java:506) ~[elasticsea
rch-7.3.1.jar:7.3.1]
Aug 30 10:31:48 elastic-stage-vyakar-master-1 elasticsearch[8339]:         at org.elasticsearch.cluster.coordination.JoinHelper$5.handleException(JoinHelper.java:360) ~[elasti
csearch-7.3.1.jar:7.3.1]
Aug 30 10:31:48 elastic-stage-vyakar-master-1 elasticsearch[8339]:         at org.elasticsearch.transport.TransportService$ContextRestoreResponseHandler.handleException(Transp
ortService.java:1111) ~[elasticsearch-7.3.1.jar:7.3.1]
Aug 30 10:31:48 elastic-stage-vyakar-master-1 elasticsearch[8339]:         at org.elasticsearch.transport.InboundHandler.lambda$handleException$2(InboundHandler.java:246) ~[el
asticsearch-7.3.1.jar:7.3.1]
Aug 30 10:31:48 elastic-stage-vyakar-master-1 elasticsearch[8339]:         at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContex
t.java:688) ~[elasticsearch-7.3.1.jar:7.3.1]
Aug 30 10:31:48 elastic-stage-vyakar-master-1 elasticsearch[8339]:         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) ~[?:?]
Aug 30 10:31:48 elastic-stage-vyakar-master-1 elasticsearch[8339]:         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) ~[?:?]
Aug 30 10:31:48 elastic-stage-vyakar-master-1 elasticsearch[8339]:         at java.lang.Thread.run(Thread.java:835) [?:?]
Aug 30 10:31:48 elastic-stage-vyakar-master-1 elasticsearch[8339]: Caused by: org.elasticsearch.transport.RemoteTransportException: [node-4][10.128.0.30:9300][internal:cluster
/coordination/join/validate]
Aug 30 10:31:48 elastic-stage-vyakar-master-1 elasticsearch[8339]: Caused by: org.elasticsearch.cluster.coordination.CoordinationStateRejectedException: join validation on clu
ster state with a different cluster uuid cBaVe_mHT7mmhgB0VBqQGw than local cluster uuid P0xgJFsgQP2NDe1Xgop_tg, rejecting
Aug 30 10:31:48 elastic-stage-vyakar-master-1 elasticsearch[8339]:         at org.elasticsearch.cluster.coordination.JoinHelper.lambda$new$4(JoinHelper.java:148) ~[elasticsear
ch-7.3.1.jar:7.3.1]
Aug 30 10:31:48 elastic-stage-vyakar-master-1 elasticsearch[8339]:         at org.elasticsearch.xpack.security.transport.SecurityServerTransportInterceptor$ProfileSecuredReque
stHandler$1.doRun(SecurityServerTransportInterceptor.java:257) ~[?:?]
Aug 30 10:31:48 elastic-stage-vyakar-master-1 elasticsearch[8339]:         at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elastic
search-7.3.1.jar:7.3.1]
Aug 30 10:31:48 elastic-stage-vyakar-master-1 elasticsearch[8339]:         at org.elasticsearch.xpack.security.transport.SecurityServerTransportInterceptor$ProfileSecuredReque
stHandler.messageReceived(SecurityServerTransportInterceptor.java:315) ~[?:?]
Aug 30 10:31:48 elastic-stage-vyakar-master-1 elasticsearch[8339]:         at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.
java:63) ~[elasticsearch-7.3.1.jar:7.3.1]
Aug 30 10:31:48 elastic-stage-vyakar-master-1 elasticsearch[8339]:         at org.elasticsearch.transport.InboundHandler$RequestHandler.doRun(InboundHandler.java:267) ~[elasti
csearch-7.3.1.jar:7.3.1]
Aug 30 10:31:48 elastic-stage-vyakar-master-1 elasticsearch[8339]:         at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(Th
readContext.java:758) ~[elasticsearch-7.3.1.jar:7.3.1]
Aug 30 10:31:48 elastic-stage-vyakar-master-1 elasticsearch[8339]:         at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elastic
search-7.3.1.jar:7.3.1]
Aug 30 10:31:48 elastic-stage-vyakar-master-1 elasticsearch[8339]:         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) ~[?:?]
Aug 30 10:31:48 elastic-stage-vyakar-master-1 elasticsearch[8339]:         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) ~[?:?]
Aug 30 10:31:48 elastic-stage-vyakar-master-1 elasticsearch[8339]:         at java.lang.Thread.run(Thread.java:835) ~[?:?]

DavidTurner · August 30, 2019, 11:14am

This node used to belong to a different cluster, and you cannot move nodes between clusters.

If this happens every time you add a new node then there is something very wrong with your process.

Ayush_Verma · August 30, 2019, 11:20am

Thanks for the reply, But how is it possible i have same cluster.name for all nodes. let me tell you the process i follow.

Create new instance install elasticsearch in it, configure yml file as i already mentioned.
For security i copy the certificate that i created for node-1 paste that certificate in /etc/elasticsearch/ for new node.
add xpack security setting in yml file, and restart all nodes.
Expect new node in the family.

DavidTurner · August 30, 2019, 11:31am

Does Elasticsearch run on the new node before you have configured it?

Ayush_Verma · August 30, 2019, 11:36am

yes i created new instance for each node of elasticsearch. Actually i am using google cloud instance. Separate instance or machine for each node i don't know is this is information matters or not but for this node i used system with 1.74GM memory and other 3 nodes are installed in 3.75 GB memory.

node-4 logs

Aug 30 11:38:19 elastic-stage-vyakar-master-1 elasticsearch[9362]: Caused by: java.lang.IllegalStateException: Node is started with node.data=false, but has shard data: [/var/
lib/elasticsearch/nodes/0/indices/1DTqsfUmRPKv2szFCXfc-Q/0]. Use 'elasticsearch-node repurpose' tool to clean up
Aug 30 11:38:19 elastic-stage-vyakar-master-1 elasticsearch[9362]:         at org.elasticsearch.env.NodeEnvironment.ensureNoShardData(NodeEnvironment.java:1081) ~[elasticsearc
h-7.3.1.jar:7.3.1]
Aug 30 11:38:19 elastic-stage-vyakar-master-1 elasticsearch[9362]:         at org.elasticsearch.env.NodeEnvironment.<init>(NodeEnvironment.java:325) ~[elasticsearch-7.3.1.jar:
7.3.1]
Aug 30 11:38:19 elastic-stage-vyakar-master-1 elasticsearch[9362]:         at org.elasticsearch.node.Node.<init>(Node.java:278) ~[elasticsearch-7.3.1.jar:7.3.1]
Aug 30 11:38:19 elastic-stage-vyakar-master-1 elasticsearch[9362]:         at org.elasticsearch.node.Node.<init>(Node.java:258) ~[elasticsearch-7.3.1.jar:7.3.1]
Aug 30 11:38:19 elastic-stage-vyakar-master-1 elasticsearch[9362]:         at org.elasticsearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:221) ~[elasticsearch-7.3.1.jar:7.3.
1]
Aug 30 11:38:19 elastic-stage-vyakar-master-1 elasticsearch[9362]:         at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:221) ~[elasticsearch-7.3.1.jar:7.3.1]
Aug 30 11:38:19 elastic-stage-vyakar-master-1 elasticsearch[9362]:         at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:349) ~[elasticsearch-7.3.1.jar:7.3.1]
Aug 30 11:38:19 elastic-stage-vyakar-master-1 elasticsearch[9362]:         at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:159) ~[elasticsearch-7.3.1.jar:
7.3.1]
Aug 30 11:38:19 elastic-stage-vyakar-master-1 elasticsearch[9362]:         ... 6 more
Aug 30 11:38:19 elastic-stage-vyakar-master-1 systemd[1]: elasticsearch.service: Main process exited, code=exited, status=1/FAILURE
Aug 30 11:38:19 elastic-stage-vyakar-master-1 systemd[1]: elasticsearch.service: Failed with result 'exit-code'.
Aug 30 11:40:35 elastic-stage-vyakar-master-1 systemd[1]: Started Elasticsearch.
Aug 30 11:40:37 elastic-stage-vyakar-master-1 elasticsearch[9506]: OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be
 removed in a future release.
Aug 30 11:40:42 elastic-stage-vyakar-master-1 elasticsearch[9506]: [2019-08-30T11:40:42,230][INFO ][o.e.e.NodeEnvironment    ] [node-4] using [1] data paths, mounts [[/ (/dev/
sda1)]], net usable_space [7.3gb], net total_space [9.5gb], types [ext4]
Aug 30 11:40:42 elastic-stage-vyakar-master-1 elasticsearch[9506]: [2019-08-30T11:40:42,243][INFO ][o.e.e.NodeEnvironment    ] [node-4] heap size [1015.6mb], compressed ordina
ry object pointers [true]
Aug 30 11:40:42 elastic-stage-vyakar-master-1 elasticsearch[9506]: [2019-08-30T11:40:42,325][WARN ][o.e.b.ElasticsearchUncaughtExceptionHandler] [node-4] uncaught exception in
 thread [main]
Aug 30 11:40:42 elastic-stage-vyakar-master-1 elasticsearch[9506]: org.elasticsearch.bootstrap.StartupException: java.lang.IllegalStateException: Node is started with node.dat
a=false, but has shard data: [/var/lib/elasticsearch/nodes/0/indices/1DTqsfUmRPKv2szFCXfc-Q/0]. Use 'elasticsearch-node repurpose' tool to clean up
Aug 30 11:40:42 elastic-stage-vyakar-master-1 elasticsearch[9506]:         at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:163) ~[elasticsearch-7.3.1.jar:
7.3.1]
Aug 30 11:40:42 elastic-stage-vyakar-master-1 elasticsearch[9506]:         at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:150) ~[elasticsearch-7.3.1.j
ar:7.3.1]
Aug 30 11:40:42 elastic-stage-vyakar-master-1 elasticsearch[9506]:         at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:86) ~[elastics
earch-7.3.1.jar:7.3.1]
Aug 30 11:40:42 elastic-stage-vyakar-master-1 elasticsearch[9506]:         at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:124) ~[elasticsearch-cli-7.3.
1.jar:7.3.1]
Aug 30 11:40:42 elastic-stage-vyakar-master-1 elasticsearch[9506]:         at org.elasticsearch.cli.Command.main(Command.java:90) ~[elasticsearch-cli-7.3.1.jar:7.3.1]
Aug 30 11:40:42 elastic-stage-vyakar-master-1 elasticsearch[9506]:         at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:115) ~[elasticsearch-7.3.1.jar:
7.3.1]

DavidTurner · August 30, 2019, 11:52am

It looks like you are trying to configure the node after it's already started up once. It's important to configure it correctly before running it for the first time.

system · September 27, 2019, 11:52am

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.