# Multiple indexes break elasticsearch (2.3.1) cluster replication?
**URL:** https://discuss.elastic.co/t/multiple-indexes-break-elasticsearch-2-3-1-cluster-replication/161689
**Category:** Elasticsearch
**Created:** [December 20, 2018, 11:41am UTC](https://discuss.elastic.co/t/multiple-indexes-break-elasticsearch-2-3-1-cluster-replication/161689 "2018-12-20T11:41:09Z")
**Posts on this page:** 6
**Page:** 1
### Author:  [@patadaptive](https://discuss.elastic.co/u/patadaptive)
#### Post date: [December 20, 2018, 11:41am UTC](https://discuss.elastic.co/t/multiple-indexes-break-elasticsearch-2-3-1-cluster-replication/161689/1 "2018-12-20T11:41:09Z")
I have a 3 node Elasticsearch cluster (2 data nodes).
My solution needs to support full replication if any 1 of the 3 nodes goes down or simply shutdown. I've been turning the ES server processes off and on and the replication always works correctly no matter what I do - when I have just 1 index (defaults of 5 primary shards and 1 replica). So everything works great and does what is required.
But a new requirement has just come in to add a second separate index (same defaults of 5 primary shards and 1 replica). But since I added this new index (even though it contains no data) it seems to break the replication on my first index. In that when I test it by shutting down and rebooting nodes I can see data go missing in kibana. Sometimes when I restore the previous master I can see the data come back.
Is this expected behavior in Elasticsearch 2.3.1 - can adding a second index break replication like this?
Thanks
---
### Author:  [@patadaptive](https://discuss.elastic.co/u/patadaptive)
#### Post date: [December 20, 2018, 1:01pm UTC](https://discuss.elastic.co/t/multiple-indexes-break-elasticsearch-2-3-1-cluster-replication/161689/2 "2018-12-20T13:01:06Z")
Ok I narrowed in further on the issue - it has nothing to do with the second index...
if I reboot the current master - its like I loose the data that was inserted in the last "few" minutes.
Stopping the process and leaving the server running is fine - there is no loss of data,
But shutting down the vm, results in the above loss of data.
---
### Author:  [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)
#### Post date: [December 20, 2018, 1:02pm UTC](https://discuss.elastic.co/t/multiple-indexes-break-elasticsearch-2-3-1-cluster-replication/161689/3 "2018-12-20T13:02:53Z")
How many master eligible nodes do you have in the cluster? If this is 2 or 3, have you set `discovery.zen.minimum_master_nodes` to `2` so you can [avoid split brain scenarios](https://www.elastic.co/guide/en/elasticsearch/reference/6.5/modules-node.html#split-brain)?
---
### Author:  [@patadaptive](https://discuss.elastic.co/u/patadaptive)
#### Post date: [December 20, 2018, 2:50pm UTC](https://discuss.elastic.co/t/multiple-indexes-break-elasticsearch-2-3-1-cluster-replication/161689/4 "2018-12-20T14:50:24Z")
Thanks for your quick reply.
My cluster has 3 eligible master nodes and:
discovery.zen.minimum\_master\_nodes is set to `2`
The issue just seems to be with the cluster node being issued a shutdown command and elasticsearch is not getting to shutdown correctly - if I call a shutdown script when the cluster node is being shutdown - it should fix the issue.
Thanks again
---
### Author:  [@patadaptive](https://discuss.elastic.co/u/patadaptive)
#### Post date: [December 21, 2018, 2:25pm UTC](https://discuss.elastic.co/t/multiple-indexes-break-elasticsearch-2-3-1-cluster-replication/161689/5 "2018-12-21T14:25:58Z")
Yes doing the following solved the issue:
On each Elasticsearch node, I edited this file:
/usr/lib/systemd/system/elasticsearch.service
And added this line:
ExecStop=/usr/share/elasticsearch/bin/elasticsearch stop
Now when a node is issued a shutdown or reboot command no data is lossed.
Thanks again
---
### Author:  [@system](https://discuss.elastic.co/u/system)
#### Post date: [January 18, 2019, 2:25pm UTC](https://discuss.elastic.co/t/multiple-indexes-break-elasticsearch-2-3-1-cluster-replication/161689/6 "2019-01-18T14:25:59Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.