# Clustering failed
**URL:** https://discuss.elastic.co/t/clustering-failed/211911
**Category:** Elasticsearch
**Created:** [December 15, 2019, 1:12pm UTC](https://discuss.elastic.co/t/clustering-failed/211911 "2019-12-15T13:12:28Z")
**Posts on this page:** 8
**Page:** 1
### Author:  [@epacke](https://discuss.elastic.co/u/epacke)
#### Post date: [December 15, 2019, 1:12pm UTC](https://discuss.elastic.co/t/clustering-failed/211911/1 "2019-12-15T13:12:29Z")
Dear experts
I've had one elasticsearch server at home for a few months now and it's been working well for a few months. However, with an power outage I lost a few indices and looking around it seems like having a cluster would have avoided at least part of the "damage done".
Thus I started with creating a two more elastic search servers, one master with no node data, and another data node. However, I have not been able to get it working so I am hoping to get some pointers here.
It looks like data has been replicated between the data nodes but the data nodes only shows as one when issuing the cluster status command. The data nodes has the same **cluster\_uuid** but the master node does not. Kibana also shows only one node even though I configure it to use all three.
Any ideas?
**Master node (no data)**
```
cluster.name: siem
node.name: siem-master
path.data: /var/lib/elasticsearch
path.logs: /var/log/elasticsearch
network.host: 192.168.70.162
http.port: 9200
discovery.seed_hosts:
- 192.168.70.150
- 192.168.70.161
- 192.168.70.162
cluster.initial_master_nodes:
- siem-master
- siem-1
- siem-2
node.data: false
```
**Data node 1**
```
cluster.name: siem
node.name: siem-1
path.data: /var/lib/elasticsearch
path.logs: /var/log/elasticsearch
network.host: 192.168.70.150
http.port: 9200
discovery.seed_hosts:
- 192.168.70.150
- 192.168.70.161
- 192.168.70.162
cluster.initial_master_nodes:
- siem-master
- siem-1
- siem-2
cluster.max_shards_per_node: 2000
```
**Data node 2**
```
cluster.name: siem
node.name: siem-2
path.data: /var/lib/elasticsearch
path.logs: /var/log/elasticsearch
network.host: 192.168.70.161
http.port: 9200
discovery.seed_hosts:
- 192.168.70.150
- 192.168.70.161
- 192.168.70.162
cluster.initial_master_nodes:
- siem-master
- siem-1
- siem-2
cluster.max_shards_per_node: 2000
```
**Cluster status output**
```
{
"cluster_name" : "siem",
"status" : "red",
"timed_out" : false,
"number_of_nodes" : 1,
"number_of_data_nodes" : 0,
"active_primary_shards" : 0,
"active_shards" : 0,
"relocating_shards" : 0,
"initializing_shards" : 0,
"unassigned_shards" : 2,
"delayed_unassigned_shards" : 0,
"number_of_pending_tasks" : 0,
"number_of_in_flight_fetch" : 0,
"task_max_waiting_in_queue_millis" : 0,
"active_shards_percent_as_number" : 0.0
}
```
**Master Node: siem-master/\_cluster/health?pretty**
```
{
"cluster_name" : "siem",
"status" : "yellow",
"timed_out" : false,
"number_of_nodes" : 1,
"number_of_data_nodes" : 1,
"active_primary_shards" : 1961,
"active_shards" : 1961,
"relocating_shards" : 0,
"initializing_shards" : 0,
"unassigned_shards" : 14,
"delayed_unassigned_shards" : 0,
"number_of_pending_tasks" : 0,
"number_of_in_flight_fetch" : 0,
"task_max_waiting_in_queue_millis" : 0,
"active_shards_percent_as_number" : 99.29113924050633
}
```
**Data Node 1: siem-2/\_cluster/health?pretty**
```
{
"cluster_name" : "siem",
"status" : "yellow",
"timed_out" : false,
"number_of_nodes" : 1,
"number_of_data_nodes" : 1,
"active_primary_shards" : 1957,
"active_shards" : 1957,
"relocating_shards" : 0,
"initializing_shards" : 0,
"unassigned_shards" : 11,
"delayed_unassigned_shards" : 0,
"number_of_pending_tasks" : 0,
"number_of_in_flight_fetch" : 0,
"task_max_waiting_in_queue_millis" : 0,
"active_shards_percent_as_number" : 99.4410569105691
}
```
---
### Author:  [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)
#### Post date: [December 15, 2019, 1:57pm UTC](https://discuss.elastic.co/t/clustering-failed/211911/2 "2019-12-15T13:57:47Z")
When adding nodes to an existing cluster you should not set `cluster.initial_master_nodes`, just `discovery.seed_hosts`, and the new nodes should have an empty data path when started for the first time.
Your config looks ok now (except all three nodes are data nodes?) but I think this wasn't the config used the first time these nodes started up. Best to start again with empty data paths for the two new nodes.
---
### Author:  [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)
#### Post date: [December 15, 2019, 2:02pm UTC](https://discuss.elastic.co/t/clustering-failed/211911/3 "2019-12-15T14:02:53Z")
Also if you suffered data loss due to a power outage then this indicates something wrong with your disks, often a volatile write cache. The proper fix is to address this misconfiguration, rather than just adding nodes. I'm guessing that you don't have redundant power feeds at your home, so a power failure will likely affect all nodes and therefore has a good chance of corrupting all copies of each active shard if your storage isn't set up for power safety.
---
### Author:  [@epacke](https://discuss.elastic.co/u/epacke)
#### Post date: [December 15, 2019, 3:28pm UTC](https://discuss.elastic.co/t/clustering-failed/211911/4 "2019-12-15T15:28:35Z")
Thank you David
I tried to do a complete re-install of Ubuntu and then installed elasticsearch again.
However, the node would not start and it looks like the culprit was the removal of the log- and data-path (looks like it defaults to /usr/share/elasticsearch/(data|logs) where it does not have write permissions.
Re-adding the path.data and path.logs configuration made it start and I now have three nodes in the cluster and indices are slowly turning from yellow to green.
You mentioned that all three nodes are data nodes but the master node had this config at the last line: **node.data: false**. Is that not sufficient?
Regarding the outage I was hoping that some of the data in one of the nodes would be intact and not corrupted and that this could be used for "self-healing" (a bit like RAID5). Perhaps a silly assumption (which most is).
Correct, no redundant power feeds in my home. I'd suggest it to my wife but I have a feeling that the result would be me and my VMware environment being left in the street with not power at all. 🙂
Thanks for all the help and merry X-mas!
**Data node 1**
```auto
cluster.name: siem
node.name: siem-1
path.data: /var/lib/elasticsearch
path.logs: /var/log/elasticsearch
network.host: 192.168.70.150
discovery.seed_hosts:
- 192.168.70.150
- 192.168.70.161
cluster.max_shards_per_node: 2000
```
**Data node 2**
```auto
cluster.name: siem
node.name: siem-2
path.logs: /var/log/elasticsearch
network.host: 192.168.70.161
discovery.seed_hosts:
- 192.168.70.150
- 192.168.70.161
cluster.max_shards_per_node: 2000
```
**Master node**
```auto
cluster.name: siem
node.name: siem-master
path.data: /var/lib/elasticsearch
path.logs: /var/log/elasticsearch
network.host: 192.168.70.162
discovery.seed_hosts:
- 192.168.70.150
- 192.168.70.161
- 192.168.70.162
node.master: true
node.voting_only: false
node.data: false
node.ingest: false
node.ml: false
xpack.ml.enabled: true
cluster.remote.connect: false
```
---
### Author:  [@epacke](https://discuss.elastic.co/u/epacke)
#### Post date: [December 15, 2019, 3:50pm UTC](https://discuss.elastic.co/t/clustering-failed/211911/5 "2019-12-15T15:50:59Z")
My take from this is that the **cluster.initial\_master\_nodes** option is used for bootstrapping (setting up totally new cluster from scratch?), while I tried to **add nodes** to a single node setup.
So removing that option seems to have been the magic to make it work?
---
### Author:  [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)
#### Post date: [December 16, 2019, 7:53am UTC](https://discuss.elastic.co/t/clustering-failed/211911/6 "2019-12-16T07:53:43Z")
> [@epacke](#):
>
> You mentioned that all three nodes are data nodes but the master node had this config at the last line: **node.data: false**. Is that not sufficient?
Ah apologies, I managed to miss that vital line. Yes, that's sufficient.
> [@epacke](#):
>
> Regarding the outage I was hoping that some of the data in one of the nodes would be intact and not corrupted and that this could be used for "self-healing" (a bit like RAID5). Perhaps a silly assumption (which most is).
This is still relying on luck, and the probability of losing is still pretty high. Best to fix your storage.
> [@epacke](#):
>
> So removing that option seems to have been the magic to make it work?
Really it was deleting the contents of the data path that did it. All your nodes had already bootstrapped their own one-node clusters and you can't merge clusters together once they've formed. But once you've got a working cluster you don't want to be accidentally bootstrapping another one, hence why we recommend not using `cluster.initial_master_nodes`.
---
### Author:  [@epacke](https://discuss.elastic.co/u/epacke)
#### Post date: [December 17, 2019, 7:03pm UTC](https://discuss.elastic.co/t/clustering-failed/211911/7 "2019-12-17T19:03:48Z")
Thank you David for the help. Much appreciated!
---
### Author:  [@system](https://discuss.elastic.co/u/system)
#### Post date: [January 14, 2020, 7:04pm UTC](https://discuss.elastic.co/t/clustering-failed/211911/8 "2020-01-14T19:04:00Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.