# Is my data really replicated across the cluster? How to know?

**URL:** <https://discuss.elastic.co/t/is-my-data-really-replicated-across-the-cluster-how-to-know/237382>\
**Category:** Elasticsearch\
**Created:** [June 17, 2020, 3:14am UTC](https://discuss.elastic.co/t/is-my-data-really-replicated-across-the-cluster-how-to-know/237382 "2020-06-17T03:14:12Z")\
**Posts on this page:** 9\
**Page:** 1

<div class="post-metadata">

**Author:** ![pszm](https://avatars.discourse-cdn.com/v4/letter/p/e36b37/32.png) [@pszm](https://discuss.elastic.co/u/pszm)\
**Post date:** [June 17, 2020, 3:14am UTC](https://discuss.elastic.co/t/is-my-data-really-replicated-across-the-cluster-how-to-know/237382/1 "2020-06-17T03:14:12Z")

</div>

We used to have just a single node in our cluster. In the past 2 weeks, we added 2 more nodes. Now we want to switch off the old node because the machine is old. But I am doubtful that all our data has actually been spread out to the other machines in the cluster.  
I can't find a post that explains how to confirm this.

I _believe_ that doing a `_cat/shards` will tell me. Here is the output from that:

```auto
$ curl -XGET 'localhost:9200/_cat/shards/our_index_1,our_index_2,our_index_3?pretty'

our_index_1 1 r STARTED 28380731 17.3gb machine-02-ip elasticsearch-02
our_index_1 1 p STARTED 28380731 17.3gb machine-03-ip elasticsearch-03
our_index_1 3 r STARTED 28378851 17gb machine-03-ip elasticsearch-03
our_index_1 3 p STARTED 28378848 17gb machine-01-ip elasticsearch-01
our_index_1 2 r STARTED 28385815 17.5gb machine-03-ip elasticsearch-03
our_index_1 2 p STARTED 28385815 17.4gb machine-01-ip elasticsearch-01
our_index_1 4 r STARTED 28370118 17gb machine-02-ip elasticsearch-02
our_index_1 4 p STARTED 28370114 16.9gb machine-03-ip elasticsearch-03
our_index_1 0 r STARTED 28378628 16.9gb machine-02-ip elasticsearch-02
our_index_1 0 p STARTED 28378628 16.9gb machine-01-ip elasticsearch-01

our_index_2 0 p STARTED 2339117 1.4gb machine-03-ip elasticsearch-03
our_index_2 0 r STARTED 2341647 1.5gb machine-01-ip elasticsearch-01

our_index_3 1 r STARTED 1928 8mb machine-03-ip elasticsearch-03
our_index_3 1 p STARTED 1928 8.5mb machine-01-ip elasticsearch-01
our_index_3 3 r STARTED 1965 6.7mb machine-03-ip elasticsearch-03
our_index_3 3 p STARTED 1965 7.5mb machine-01-ip elasticsearch-01
our_index_3 2 r STARTED 2011 7mb machine-02-ip elasticsearch-02
our_index_3 2 p STARTED 2011 8.5mb machine-03-ip elasticsearch-03
our_index_3 4 p STARTED 2049 6.5mb machine-02-ip elasticsearch-02
our_index_3 4 r STARTED 2049 6.6mb machine-03-ip elasticsearch-03
our_index_3 0 r STARTED 1956 7.7mb machine-02-ip elasticsearch-02
our_index_3 0 p STARTED 1956 9.2mb machine-01-ip elasticsearch-01

```

It looks like all shards are on 2 machines, but no shards are on 3. So in theory if I turn off machine 1 all data will only be on 1 machine (either machine 2 or machine 3).  
Is this correct? I am not sure it is what we want. Is this a configuration issue?

---

<div class="post-metadata">

**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\
**Post date:** [June 17, 2020, 5:12am UTC](https://discuss.elastic.co/t/is-my-data-really-replicated-across-the-cluster-how-to-know/237382/2 "2020-06-17T05:12:19Z")

</div>

There's a heap on the 3rd node, here's just some of them.;

```
our_index_1 1 p STARTED 28380731 17.3gb machine-03-ip elasticsearch-03
our_index_1 3 r STARTED 28378851 17gb machine-03-ip elasticsearch-03
our_index_1 2 r STARTED 28385815 17.5gb machine-03-ip elasticsearch-03

```

If you cluster is green, and every index has a single replica set, then you are fine.

---

<div class="post-metadata">

**Author:** ![pszm](https://avatars.discourse-cdn.com/v4/letter/p/e36b37/32.png) [@pszm](https://discuss.elastic.co/u/pszm)\
**Post date:** [June 17, 2020, 5:55am UTC](https://discuss.elastic.co/t/is-my-data-really-replicated-across-the-cluster-how-to-know/237382/3 "2020-06-17T05:55:38Z")

</div>

Thanks. I guess I was expecting all shards to have 3 entries/copies - 1 on each machine. But it seems that it is done just with 2 copies.

---

<div class="post-metadata">

**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\
**Post date:** [June 17, 2020, 5:57am UTC](https://discuss.elastic.co/t/is-my-data-really-replicated-across-the-cluster-how-to-know/237382/4 "2020-06-17T05:57:23Z")

</div>

The primary, plus N replicas. In your case N =1, which is the default.

If you want every node to have a copy then increase the replica count to 2. Then there will be 3 copies, the primary plus 2 replicas.

---

<div class="post-metadata">

**Author:** ![pszm](https://avatars.discourse-cdn.com/v4/letter/p/e36b37/32.png) [@pszm](https://discuss.elastic.co/u/pszm)\
**Post date:** [June 17, 2020, 6:00am UTC](https://discuss.elastic.co/t/is-my-data-really-replicated-across-the-cluster-how-to-know/237382/5 "2020-06-17T06:00:24Z")

</div>

Ah! Thanks!

---

<div class="post-metadata">

**Author:** ![pszm](https://avatars.discourse-cdn.com/v4/letter/p/e36b37/32.png) [@pszm](https://discuss.elastic.co/u/pszm)\
**Post date:** [June 17, 2020, 6:17am UTC](https://discuss.elastic.co/t/is-my-data-really-replicated-across-the-cluster-how-to-know/237382/6 "2020-06-17T06:17:33Z")

</div>

To follow up on this. I thought that to 'turn off' one of the nodes (`machine-01`, which happened to be the master node) I simply needed to change `cluster.initial_master_nodes` and `discover.seed_hosts` on `machine-02` to be itself as opposed to being `machine-01`.  
`machine-03` already has `machine-02` as the initial master and seed host.

But when I do that, `machine-02` is no longer part of the cluster:

```auto
$ curl -XGET 'localhost:9200/?pretty'
{
  "name" : "elasticsearch-02",
  "cluster_name" : "zm-amz-data",
  "cluster_uuid" : "_na_",
  "version" : {
    "number" : "7.7.1",
    "build_flavor" : "oss",
    "build_type" : "deb",
    "build_hash" : "ad56dce891c901a492bb1ee393f12dfff473a423",
    "build_date" : "2020-05-28T16:30:01.040088Z",
    "build_snapshot" : false,
    "lucene_version" : "8.5.1",
    "minimum_wire_compatibility_version" : "6.8.0",
    "minimum_index_compatibility_version" : "6.0.0-beta1"
  },
  "tagline" : "You Know, for Search"
}

```

`machine-02 elasticsearch.yml`:

```auto
discovery.seed_hosts: ["machine-02-ip"]
cluster.initial_master_nodes: ["elasticsearch-02"]

```

`machine-03 elasticsearch.yml`:

```auto
discovery.seed_hosts: ["machine-02-ip"]
cluster.initial_master_nodes: ["elasticsearch-02"]

```

---

<div class="post-metadata">

**Author:** ![defalt](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/defalt/32/71379_2.png) [@defalt](https://discuss.elastic.co/u/defalt)\
**Post date:** [June 17, 2020, 7:20am UTC](https://discuss.elastic.co/t/is-my-data-really-replicated-across-the-cluster-how-to-know/237382/7 "2020-06-17T07:20:58Z")

</div>

It is no longer part of the cluster because it doesnt find the other nodes i guess. Because

```auto
discovery.seed_hosts: ["machine-02-ip"]

```

leads to the problem that it only knows its own IP and doesnt know where all the other nodes are at.

You should feed all IP's into discovery.seed\_hosts:

```auto
discovery.seed_hosts: ["machine-01-ip","machine-02-ip","machine-03-ip"]

```

For a small cluster its finde to set all nodes to node.master: true and register them with cluster.initial\_master\_node. By this they become master eligible which helpes with turning off and on nodes. This is the setup that works best for me.

---

<div class="post-metadata">

**Author:** ![DavidTurner](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/davidturner/32/22453_2.png) [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)\
**Post date:** [June 17, 2020, 7:32am UTC](https://discuss.elastic.co/t/is-my-data-really-replicated-across-the-cluster-how-to-know/237382/8 "2020-06-17T07:32:38Z")

</div>

You should not be changing `cluster.initial_master_nodes` after the cluster has formed. Simply remove it entirely from the config file. Quoting [the docs](https://www.elastic.co/guide/en/elasticsearch/reference/current/discovery-settings.html#initial_master_nodes):

> You should not use this setting when restarting a cluster or adding a new node to an existing cluster.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [July 15, 2020, 7:32am UTC](https://discuss.elastic.co/t/is-my-data-really-replicated-across-the-cluster-how-to-know/237382/9 "2020-07-15T07:32:45Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.