# Unable to add new node in 3 node cluster

**URL:** <https://discuss.elastic.co/t/unable-to-add-new-node-in-3-node-cluster/197490>\
**Category:** Elasticsearch\
**Created:** [August 30, 2019, 10:22am UTC](https://discuss.elastic.co/t/unable-to-add-new-node-in-3-node-cluster/197490 "2019-08-30T10:22:55Z")\
**Posts on this page:** 8\
**Page:** 1

<div class="post-metadata">

**Author:** ![Ayush\_Verma](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ayush_verma/32/46709_2.png) [@Ayush\_Verma](https://discuss.elastic.co/u/Ayush_Verma)\
**Post date:** [August 30, 2019, 10:22am UTC](https://discuss.elastic.co/t/unable-to-add-new-node-in-3-node-cluster/197490/1 "2019-08-30T10:22:55Z")

</div>

I have elasticsearch 7.3 installed in 4 nodes, i have successfully configured 3 node cluster but the thing that disturbs me is every time i want to add new node i have to delete old data (i.e /var/lib/elasticsearch/) then after restarting nodes i got the new node in the picture. So this time also i want to add new node but don't want to delete old data, is it because of security, also i am not able to set password for node-4.

**es.yml node-1**

```
cluster.name: ElasticsearchStaging
node.name: node-1
path.data: /var/lib/elasticsearch
network.host: 10.128.0.26 
http.port: 9200
discovery.seed_hosts:
   - 10.128.0.26
   - 10.128.0.24
   - 10.128.0.27
   - 10.128.0.30
cluster.initial_master_nodes:
   - node-1
   - node-2
   - node-3
   - node-4
xpack.security.enabled: true
xpack.security.transport.ssl.enabled: true
xpack.security.transport.ssl.verification_mode: certificate
xpack.security.transport.ssl.keystore.path: elastic-certificates.p12
xpack.security.transport.ssl.truststore.path: elastic-certificates.p12

```

**es.yml node-2**

```
cluster.name: ElasticsearchStaging
node.name: node-2
path.data: /var/lib/elasticsearch
network.host: 10.128.0.24 
http.port: 9200
discovery.seed_hosts:
   - 10.128.0.26
   - 10.128.0.24
   - 10.128.0.27
   - 10.128.0.30
cluster.initial_master_nodes:
   - node-1
   - node-2
   - node-3
   - node-4
xpack.security.enabled: true
xpack.security.transport.ssl.enabled: true
xpack.security.transport.ssl.verification_mode: certificate
xpack.security.transport.ssl.keystore.path: elastic-certificates.p12
xpack.security.transport.ssl.truststore.path: elastic-certificates.p12

```

**es.yml node-3**

```
cluster.name: ElasticsearchStaging
node.name: node-3
path.data: /var/lib/elasticsearch
network.host: 10.128.0.27
http.port: 9200
discovery.seed_hosts:
   - 10.128.0.26
   - 10.128.0.24
   - 10.128.0.27
   - 10.128.0.30
cluster.initial_master_nodes:
   - node-1
   - node-2
   - node-3
   - node-4
xpack.security.enabled: true
xpack.security.transport.ssl.enabled: true
xpack.security.transport.ssl.verification_mode: certificate
xpack.security.transport.ssl.keystore.path: elastic-certificates.p12
xpack.security.transport.ssl.truststore.path: elastic-certificates.p12

```

**es.yml node-4**

```
cluster.name: ElasticsearchStaging
node.name: node-4
node.master: true
node.data: false
path.data: /var/lib/elasticsearch
network.host: 10.128.0.30 
http.port: 9200
discovery.seed_hosts:
   - 10.128.0.26
   - 10.128.0.24
   - 10.128.0.27
   - 10.128.0.30
cluster.initial_master_nodes:
   - node-1
   - node-2
   - node-3
   - node-4
xpack.security.enabled: true
xpack.security.transport.ssl.enabled: true
xpack.security.transport.ssl.verification_mode: certificate
xpack.security.transport.ssl.keystore.path: elastic-certificates.p12
xpack.security.transport.ssl.truststore.path: elastic-certificates.p12

```

**update i removed comment all xpack security for now in all yml files**

Thanks.

---

<div class="post-metadata">

**Author:** ![Ayush\_Verma](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ayush_verma/32/46709_2.png) [@Ayush\_Verma](https://discuss.elastic.co/u/Ayush_Verma)\
**Post date:** [August 30, 2019, 10:38am UTC](https://discuss.elastic.co/t/unable-to-add-new-node-in-3-node-cluster/197490/2 "2019-08-30T10:38:27Z")

</div>

**node-4 logs**

```
Aug 30 10:31:48 elastic-stage-vyakar-master-1 elasticsearch[8339]: [2019-08-30T10:31:48,095][INFO][o.e.c.c.JoinHelper] [node-4] failed to join {node-2}{Qz0zf9HPTiKZPiX
fBthUdg}{U5TsaIGqQnemP9PQ-48QHg}{10.128.0.24}{10.128.0.24:9300}{dim}{ml.machine_memory=3872587776, ml.max_open_jobs=20, xpack.installed=true} with JoinRequest{sourceNode={node
-4}{VWDFHsUXRdiR380f8TjQEg}{NwL7cQMhTnGnHbNtt_dyBw}{10.128.0.30}{10.128.0.30:9300}{im}{ml.machine_memory=1771679744, xpack.installed=true, ml.max_open_jobs=20}, optionalJoin=O
ptional.empty}
Aug 30 10:31:48 elastic-stage-vyakar-master-1 elasticsearch[8339]: org.elasticsearch.transport.RemoteTransportException: [node-2][10.128.0.24:9300][internal:cluster/coordinati
on/join]
Aug 30 10:31:48 elastic-stage-vyakar-master-1 elasticsearch[8339]: Caused by: java.lang.IllegalStateException: failure when sending a validation request to node
Aug 30 10:31:48 elastic-stage-vyakar-master-1 elasticsearch[8339]: at org.elasticsearch.cluster.coordination.Coordinator$2.onFailure(Coordinator.java:506) ~[elasticsea
rch-7.3.1.jar:7.3.1]
Aug 30 10:31:48 elastic-stage-vyakar-master-1 elasticsearch[8339]: at org.elasticsearch.cluster.coordination.JoinHelper$5.handleException(JoinHelper.java:360) ~[elasti
csearch-7.3.1.jar:7.3.1]
Aug 30 10:31:48 elastic-stage-vyakar-master-1 elasticsearch[8339]: at org.elasticsearch.transport.TransportService$ContextRestoreResponseHandler.handleException(Transp
ortService.java:1111) ~[elasticsearch-7.3.1.jar:7.3.1]
Aug 30 10:31:48 elastic-stage-vyakar-master-1 elasticsearch[8339]: at org.elasticsearch.transport.InboundHandler.lambda$handleException$2(InboundHandler.java:246) ~[el
asticsearch-7.3.1.jar:7.3.1]
Aug 30 10:31:48 elastic-stage-vyakar-master-1 elasticsearch[8339]: at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContex
t.java:688) ~[elasticsearch-7.3.1.jar:7.3.1]
Aug 30 10:31:48 elastic-stage-vyakar-master-1 elasticsearch[8339]: at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) ~[?:?]
Aug 30 10:31:48 elastic-stage-vyakar-master-1 elasticsearch[8339]: at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) ~[?:?]
Aug 30 10:31:48 elastic-stage-vyakar-master-1 elasticsearch[8339]: at java.lang.Thread.run(Thread.java:835) [?:?]
Aug 30 10:31:48 elastic-stage-vyakar-master-1 elasticsearch[8339]: Caused by: org.elasticsearch.transport.RemoteTransportException: [node-4][10.128.0.30:9300][internal:cluster
/coordination/join/validate]
Aug 30 10:31:48 elastic-stage-vyakar-master-1 elasticsearch[8339]: Caused by: org.elasticsearch.cluster.coordination.CoordinationStateRejectedException: join validation on clu
ster state with a different cluster uuid cBaVe_mHT7mmhgB0VBqQGw than local cluster uuid P0xgJFsgQP2NDe1Xgop_tg, rejecting
Aug 30 10:31:48 elastic-stage-vyakar-master-1 elasticsearch[8339]: at org.elasticsearch.cluster.coordination.JoinHelper.lambda$new$4(JoinHelper.java:148) ~[elasticsear
ch-7.3.1.jar:7.3.1]
Aug 30 10:31:48 elastic-stage-vyakar-master-1 elasticsearch[8339]: at org.elasticsearch.xpack.security.transport.SecurityServerTransportInterceptor$ProfileSecuredReque
stHandler$1.doRun(SecurityServerTransportInterceptor.java:257) ~[?:?]
Aug 30 10:31:48 elastic-stage-vyakar-master-1 elasticsearch[8339]: at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elastic
search-7.3.1.jar:7.3.1]
Aug 30 10:31:48 elastic-stage-vyakar-master-1 elasticsearch[8339]: at org.elasticsearch.xpack.security.transport.SecurityServerTransportInterceptor$ProfileSecuredReque
stHandler.messageReceived(SecurityServerTransportInterceptor.java:315) ~[?:?]
Aug 30 10:31:48 elastic-stage-vyakar-master-1 elasticsearch[8339]: at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.
java:63) ~[elasticsearch-7.3.1.jar:7.3.1]
Aug 30 10:31:48 elastic-stage-vyakar-master-1 elasticsearch[8339]: at org.elasticsearch.transport.InboundHandler$RequestHandler.doRun(InboundHandler.java:267) ~[elasti
csearch-7.3.1.jar:7.3.1]
Aug 30 10:31:48 elastic-stage-vyakar-master-1 elasticsearch[8339]: at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(Th
readContext.java:758) ~[elasticsearch-7.3.1.jar:7.3.1]
Aug 30 10:31:48 elastic-stage-vyakar-master-1 elasticsearch[8339]: at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elastic
search-7.3.1.jar:7.3.1]
Aug 30 10:31:48 elastic-stage-vyakar-master-1 elasticsearch[8339]: at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) ~[?:?]
Aug 30 10:31:48 elastic-stage-vyakar-master-1 elasticsearch[8339]: at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) ~[?:?]
Aug 30 10:31:48 elastic-stage-vyakar-master-1 elasticsearch[8339]: at java.lang.Thread.run(Thread.java:835) ~[?:?]
```

---

<div class="post-metadata">

**Author:** ![DavidTurner](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/davidturner/32/22453_2.png) [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)\
**Post date:** [August 30, 2019, 11:14am UTC](https://discuss.elastic.co/t/unable-to-add-new-node-in-3-node-cluster/197490/3 "2019-08-30T11:14:12Z")

</div>

> [@Ayush\_Verma](#):
>
> `join validation on cluster state with a different cluster uuid cBaVe_mHT7mmhgB0VBqQGw than local cluster uuid P0xgJFsgQP2NDe1Xgop_tg, rejecting`

This node used to belong to a different cluster, and you cannot move nodes between clusters.

If this happens every time you add a new node then there is something very wrong with your process.

---

<div class="post-metadata">

**Author:** ![Ayush\_Verma](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ayush_verma/32/46709_2.png) [@Ayush\_Verma](https://discuss.elastic.co/u/Ayush_Verma)\
**Post date:** [August 30, 2019, 11:20am UTC](https://discuss.elastic.co/t/unable-to-add-new-node-in-3-node-cluster/197490/4 "2019-08-30T11:20:16Z")

</div>

Thanks for the reply, But how is it possible i have same `cluster.name` for all nodes. let me tell you the process i follow.

1. Create new instance install elasticsearch in it, configure yml file as i already mentioned.
2. For security i copy the certificate that i created for node-1 paste that certificate in `/etc/elasticsearch/` for new node.
3. add xpack security setting in yml file, and restart all nodes.
4. Expect new node in the family.

---

<div class="post-metadata">

**Author:** ![DavidTurner](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/davidturner/32/22453_2.png) [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)\
**Post date:** [August 30, 2019, 11:31am UTC](https://discuss.elastic.co/t/unable-to-add-new-node-in-3-node-cluster/197490/5 "2019-08-30T11:31:20Z")

</div>

Does Elasticsearch run on the new node before you have configured it?

---

<div class="post-metadata">

**Author:** ![Ayush\_Verma](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ayush_verma/32/46709_2.png) [@Ayush\_Verma](https://discuss.elastic.co/u/Ayush_Verma)\
**Post date:** [August 30, 2019, 11:36am UTC](https://discuss.elastic.co/t/unable-to-add-new-node-in-3-node-cluster/197490/6 "2019-08-30T11:36:20Z")

</div>

yes i created new instance for each node of elasticsearch. Actually i am using google cloud instance. Separate instance or machine for each node i don't know is this is information matters or not but for this node i used system with 1.74GM memory and other 3 nodes are installed in 3.75 GB memory.

**node-4 logs**

```
Aug 30 11:38:19 elastic-stage-vyakar-master-1 elasticsearch[9362]: Caused by: java.lang.IllegalStateException: Node is started with node.data=false, but has shard data: [/var/
lib/elasticsearch/nodes/0/indices/1DTqsfUmRPKv2szFCXfc-Q/0]. Use 'elasticsearch-node repurpose' tool to clean up
Aug 30 11:38:19 elastic-stage-vyakar-master-1 elasticsearch[9362]: at org.elasticsearch.env.NodeEnvironment.ensureNoShardData(NodeEnvironment.java:1081) ~[elasticsearc
h-7.3.1.jar:7.3.1]
Aug 30 11:38:19 elastic-stage-vyakar-master-1 elasticsearch[9362]: at org.elasticsearch.env.NodeEnvironment.<init>(NodeEnvironment.java:325) ~[elasticsearch-7.3.1.jar:
7.3.1]
Aug 30 11:38:19 elastic-stage-vyakar-master-1 elasticsearch[9362]: at org.elasticsearch.node.Node.<init>(Node.java:278) ~[elasticsearch-7.3.1.jar:7.3.1]
Aug 30 11:38:19 elastic-stage-vyakar-master-1 elasticsearch[9362]: at org.elasticsearch.node.Node.<init>(Node.java:258) ~[elasticsearch-7.3.1.jar:7.3.1]
Aug 30 11:38:19 elastic-stage-vyakar-master-1 elasticsearch[9362]: at org.elasticsearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:221) ~[elasticsearch-7.3.1.jar:7.3.
1]
Aug 30 11:38:19 elastic-stage-vyakar-master-1 elasticsearch[9362]: at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:221) ~[elasticsearch-7.3.1.jar:7.3.1]
Aug 30 11:38:19 elastic-stage-vyakar-master-1 elasticsearch[9362]: at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:349) ~[elasticsearch-7.3.1.jar:7.3.1]
Aug 30 11:38:19 elastic-stage-vyakar-master-1 elasticsearch[9362]: at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:159) ~[elasticsearch-7.3.1.jar:
7.3.1]
Aug 30 11:38:19 elastic-stage-vyakar-master-1 elasticsearch[9362]: ... 6 more
Aug 30 11:38:19 elastic-stage-vyakar-master-1 systemd[1]: elasticsearch.service: Main process exited, code=exited, status=1/FAILURE
Aug 30 11:38:19 elastic-stage-vyakar-master-1 systemd[1]: elasticsearch.service: Failed with result 'exit-code'.
Aug 30 11:40:35 elastic-stage-vyakar-master-1 systemd[1]: Started Elasticsearch.
Aug 30 11:40:37 elastic-stage-vyakar-master-1 elasticsearch[9506]: OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be
 removed in a future release.
Aug 30 11:40:42 elastic-stage-vyakar-master-1 elasticsearch[9506]: [2019-08-30T11:40:42,230][INFO][o.e.e.NodeEnvironment] [node-4] using [1] data paths, mounts [[/ (/dev/
sda1)]], net usable_space [7.3gb], net total_space [9.5gb], types [ext4]
Aug 30 11:40:42 elastic-stage-vyakar-master-1 elasticsearch[9506]: [2019-08-30T11:40:42,243][INFO][o.e.e.NodeEnvironment] [node-4] heap size [1015.6mb], compressed ordina
ry object pointers [true]
Aug 30 11:40:42 elastic-stage-vyakar-master-1 elasticsearch[9506]: [2019-08-30T11:40:42,325][WARN][o.e.b.ElasticsearchUncaughtExceptionHandler] [node-4] uncaught exception in
 thread [main]
Aug 30 11:40:42 elastic-stage-vyakar-master-1 elasticsearch[9506]: org.elasticsearch.bootstrap.StartupException: java.lang.IllegalStateException: Node is started with node.dat
a=false, but has shard data: [/var/lib/elasticsearch/nodes/0/indices/1DTqsfUmRPKv2szFCXfc-Q/0]. Use 'elasticsearch-node repurpose' tool to clean up
Aug 30 11:40:42 elastic-stage-vyakar-master-1 elasticsearch[9506]: at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:163) ~[elasticsearch-7.3.1.jar:
7.3.1]
Aug 30 11:40:42 elastic-stage-vyakar-master-1 elasticsearch[9506]: at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:150) ~[elasticsearch-7.3.1.j
ar:7.3.1]
Aug 30 11:40:42 elastic-stage-vyakar-master-1 elasticsearch[9506]: at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:86) ~[elastics
earch-7.3.1.jar:7.3.1]
Aug 30 11:40:42 elastic-stage-vyakar-master-1 elasticsearch[9506]: at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:124) ~[elasticsearch-cli-7.3.
1.jar:7.3.1]
Aug 30 11:40:42 elastic-stage-vyakar-master-1 elasticsearch[9506]: at org.elasticsearch.cli.Command.main(Command.java:90) ~[elasticsearch-cli-7.3.1.jar:7.3.1]
Aug 30 11:40:42 elastic-stage-vyakar-master-1 elasticsearch[9506]: at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:115) ~[elasticsearch-7.3.1.jar:
7.3.1]
```

---

<div class="post-metadata">

**Author:** ![DavidTurner](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/davidturner/32/22453_2.png) [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)\
**Post date:** [August 30, 2019, 11:52am UTC](https://discuss.elastic.co/t/unable-to-add-new-node-in-3-node-cluster/197490/7 "2019-08-30T11:52:13Z")

</div>

It looks like you are trying to configure the node after it's already started up once. It's important to configure it correctly before running it for the first time.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [September 27, 2019, 11:52am UTC](https://discuss.elastic.co/t/unable-to-add-new-node-in-3-node-cluster/197490/8 "2019-09-27T11:52:18Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.